[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 519513166 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002324] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.007480] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009009] pid_max: default: 32768 minimum: 301 [ 0.010122] LSM: Security Framework initializing [ 0.011038] Yama: becoming mindful. [ 0.012028] SELinux: Initializing. [ 0.013048] *** VALIDATE selinux *** [ 0.021380] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026204] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027137] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029100] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030102] *** VALIDATE tmpfs *** [ 0.032245] *** VALIDATE proc *** [ 0.033200] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.036228] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039023] Spectre V2 : User space: Vulnerable [ 0.040006] Speculative Store Bypass: Vulnerable [ 0.043245] debug: unmapping init [mem 0xffffffffb9859000-0xffffffffb9860fff] [ 0.045923] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046721] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047029] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050017] ... value mask: 0000ffffffffffff [ 0.051014] ... max period: 00007fffffffffff [ 0.052012] ... fixed-purpose events: 3 [ 0.053010] ... event mask: 000000070000000f [ 0.054266] rcu: Hierarchical SRCU implementation. [ 0.056454] smp: Bringing up secondary CPUs ... [ 0.057514] x86: Booting SMP configuration: [ 0.058016] .... node #0, CPUs: #1 #2 #3 [ 0.062102] smp: Brought up 1 node, 4 CPUs [ 0.064009] smpboot: Max logical packages: 1 [ 0.065017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143022] node 0 deferred pages initialised in 75ms [ 0.146228] devtmpfs: initialized [ 0.147214] x86/mm: Memory block size: 128MB [ 0.149837] gcov: version magic: 0x41383552 [ 0.152188] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.157265] pinctrl core: initialized pinctrl subsystem [ 0.159127] [ 0.159720] ************************************************************* [ 0.162011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165009] ** ** [ 0.167008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.169009] ** ** [ 0.171010] ** This means that this kernel is built to expose internal ** [ 0.173010] ** IOMMU data structures, which may compromise security on ** [ 0.175008] ** your system. ** [ 0.178013] ** ** [ 0.180009] ** If you see this message and you are not debugging the ** [ 0.183009] ** kernel, report this immediately to your vendor! ** [ 0.185010] ** ** [ 0.187007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.190009] ************************************************************* [ 0.192625] NET: Registered protocol family 16 [ 0.194381] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.197052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.200052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.203392] cpuidle: using governor menu [ 0.204482] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.206606] PCI: Using configuration type 1 for base access [ 0.208119] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.217124] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.219080] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.222070] cryptd: max_cpu_qlen set to 1000 [ 0.224228] ACPI: Added _OSI(Module Device) [ 0.226010] ACPI: Added _OSI(Processor Device) [ 0.227007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.228008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.232258] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.238303] ACPI: Interpreter enabled [ 0.240046] ACPI: PM: (supports S0 S3 S4 S5) [ 0.242010] ACPI: Using IOAPIC for interrupt routing [ 0.243083] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.247388] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.258319] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.260040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.262015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.266148] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.271385] acpiphp: Slot [2] registered [ 0.273112] acpiphp: Slot [5] registered [ 0.274121] acpiphp: Slot [6] registered [ 0.276093] acpiphp: Slot [7] registered [ 0.277093] acpiphp: Slot [8] registered [ 0.279111] acpiphp: Slot [9] registered [ 0.280122] acpiphp: Slot [10] registered [ 0.282138] acpiphp: Slot [3] registered [ 0.284092] acpiphp: Slot [4] registered [ 0.285079] acpiphp: Slot [11] registered [ 0.287085] acpiphp: Slot [12] registered [ 0.289093] acpiphp: Slot [13] registered [ 0.290096] acpiphp: Slot [14] registered [ 0.291074] acpiphp: Slot [15] registered [ 0.293077] acpiphp: Slot [16] registered [ 0.295085] acpiphp: Slot [17] registered [ 0.296101] acpiphp: Slot [18] registered [ 0.298089] acpiphp: Slot [19] registered [ 0.299064] acpiphp: Slot [20] registered [ 0.301085] acpiphp: Slot [21] registered [ 0.302086] acpiphp: Slot [22] registered [ 0.304073] acpiphp: Slot [23] registered [ 0.306076] acpiphp: Slot [24] registered [ 0.307056] acpiphp: Slot [25] registered [ 0.309080] acpiphp: Slot [26] registered [ 0.311101] acpiphp: Slot [27] registered [ 0.313095] acpiphp: Slot [28] registered [ 0.315144] acpiphp: Slot [29] registered [ 0.317097] acpiphp: Slot [30] registered [ 0.319079] acpiphp: Slot [31] registered [ 0.320153] PCI host bridge to bus 0000:00 [ 0.322026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.324018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.327017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.329017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.332015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.334014] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.336148] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.338895] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.342162] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.351012] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.356590] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.359014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.361013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.363014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.367014] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.369889] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.373042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.376885] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.382012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.396023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.402014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.409375] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.431013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.448013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.499012] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.511826] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.525013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.536014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.569014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.584085] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.598023] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.612031] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.632021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.642000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.651012] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.666014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.704014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.723569] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.734013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.747014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.782016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.793846] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.809014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.822017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.853015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.869889] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.872356] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.875356] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.879374] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.882193] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.887098] iommu: Default domain type: Passthrough [ 0.889579] SCSI subsystem initialized [ 0.891102] ACPI: bus type USB registered [ 0.893100] usbcore: registered new interface driver usbfs [ 0.895062] usbcore: registered new interface driver hub [ 0.897118] usbcore: registered new device driver usb [ 0.899275] pps_core: LinuxPPS API ver. 1 registered [ 0.901010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.905071] PTP clock support registered [ 0.907080] EDAC MC: Ver: 3.0.0 [ 0.909110] PCI: Using ACPI for IRQ routing [ 0.911292] NetLabel: Initializing [ 0.912007] NetLabel: domain hash size = 128 [ 0.914008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.916075] NetLabel: unlabeled traffic allowed by default [ 0.918250] vgaarb: loaded [ 0.919403] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.922012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.931426] clocksource: Switched to clocksource kvm-clock [ 1.040599] VFS: Disk quotas dquot_6.6.0 [ 1.041807] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.043728] *** VALIDATE ramfs *** [ 1.044807] *** VALIDATE hugetlbfs *** [ 1.046412] pnp: PnP ACPI init [ 1.048649] pnp: PnP ACPI: found 6 devices [ 1.064042] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.067254] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.068953] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.070656] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.072511] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.074402] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.076585] NET: Registered protocol family 2 [ 1.078603] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.082836] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.085766] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.090043] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.093711] TCP: Hash tables configured (established 65536 bind 65536) [ 1.097296] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.101116] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.104573] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.107524] NET: Registered protocol family 1 [ 1.110448] RPC: Registered named UNIX socket transport module. [ 1.113282] RPC: Registered udp transport module. [ 1.115372] RPC: Registered tcp transport module. [ 1.117528] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.120278] NET: Registered protocol family 44 [ 1.121539] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.123028] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.125179] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.127693] PCI: CLS 0 bytes, default 64 [ 1.129575] Unpacking initramfs... [ 2.599347] debug: unmapping init [mem 0xffff906f7cc54000-0xffff906f7ffbffff] [ 2.603240] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.605499] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.607396] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.157601] Initialise system trusted keyrings [ 3.159983] Key type blacklist registered [ 3.161699] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.170898] zbud: loaded [ 3.175358] *** VALIDATE nfs *** [ 3.176769] *** VALIDATE nfs4 *** [ 3.178333] pstore: using deflate compression [ 3.181933] Platform Keyring initialized [ 3.281439] NET: Registered protocol family 38 [ 3.282831] Key type asymmetric registered [ 3.284135] Asymmetric key parser 'x509' registered [ 3.285770] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.288648] io scheduler mq-deadline registered [ 3.290093] io scheduler kyber registered [ 3.291610] io scheduler bfq registered [ 3.293475] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.296278] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.298658] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.301137] ACPI: Power Button [PWRF] [ 3.306458] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.312754] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.351720] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.372713] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.411395] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.440587] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.469240] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.476131] Non-volatile memory driver v1.3 [ 3.477817] Linux agpgart interface v0.103 [ 3.505929] virtio_blk virtio1: [vda] 146168 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.508701] vda: detected capacity change from 0 to 74838016 [ 3.528987] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.531872] vdb: detected capacity change from 0 to 1073741824 [ 3.551299] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.554575] vdc: detected capacity change from 0 to 2621440000 [ 3.573075] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.576486] vdd: detected capacity change from 0 to 2621440000 [ 3.597647] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.601387] vde: detected capacity change from 0 to 4294967296 [ 3.623413] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.626808] vdf: detected capacity change from 0 to 4294967296 [ 3.638892] libphy: Fixed MDIO Bus: probed [ 3.644771] usbcore: registered new interface driver usbserial_generic [ 3.647570] usbserial: USB Serial support registered for generic [ 3.650150] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.654772] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.656934] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.658882] mousedev: PS/2 mouse device common for all mice [ 3.661474] rtc_cmos 00:05: RTC can wake from S4 [ 3.664599] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.664685] rtc_cmos 00:05: registered as rtc0 [ 3.670270] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.672939] intel_pstate: CPU model not supported [ 3.676301] hid: raw HID events driver (C) Jiri Kosina [ 3.678215] usbcore: registered new interface driver usbhid [ 3.678729] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.680333] usbhid: USB HID core driver [ 3.680547] drop_monitor: Initializing network drop monitor service [ 3.687678] Initializing XFRM netlink socket [ 3.689579] NET: Registered protocol family 10 [ 3.690389] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.692765] Segment Routing with IPv6 [ 3.696027] NET: Registered protocol family 17 [ 3.698406] mpls_gso: MPLS GSO support [ 3.704190] RAS: Correctable Errors collector initialized. [ 3.706076] AVX version of gcm_enc/dec engaged. [ 3.707648] AES CTR mode by8 optimization enabled [ 3.783266] sched_clock: Marking stable (3783243639, 0)->(4664041997, -880798358) [ 3.787125] registered taskstats version 1 [ 3.789282] Loading compiled-in X.509 certificates [ 3.791541] zswap: loaded using pool lzo/zbud [ 3.817405] Key type big_key registered [ 3.830956] Key type encrypted registered [ 3.832531] ima: No TPM chip found, activating TPM-bypass! [ 3.834458] ima: Allocated hash algorithm: sha1 [ 3.835666] ima: No architecture policies found [ 3.837222] evm: Initialising EVM extended attributes: [ 3.839146] evm: security.selinux [ 3.840518] evm: security.ima [ 3.841654] evm: security.capability [ 3.842975] evm: HMAC attrs: 0x1 [ 3.846195] rtc_cmos 00:05: setting system clock to 2026-09-01 13:26:46 UTC (1788269206) [ 3.852543] debug: unmapping init [mem 0xffffffffba803000-0xffffffffba9fffff] [ 3.855918] debug: unmapping init [mem 0xffffffffb9582000-0xffffffffb9858fff] [ 3.865068] Write protecting the kernel read-only data: 28672k [ 3.868469] debug: unmapping init [mem 0xffffffffb7c03000-0xffffffffb7dfffff] [ 3.871611] debug: unmapping init [mem 0xffffffffb8514000-0xffffffffb85fffff] [ 3.908241] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.917356] systemd[1]: Detected virtualization kvm. [ 3.918722] systemd[1]: Detected architecture x86-64. [ 3.920527] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.944529] systemd[1]: No hostname configured. [ 3.945894] systemd[1]: Set hostname to . [ 3.947237] random: systemd: uninitialized urandom read (16 bytes read) [ 3.948917] systemd[1]: Initializing machine ID from random generator. [ 3.997479] random: ln: uninitialized urandom read (6 bytes read) [ 4.070888] random: systemd: uninitialized urandom read (16 bytes read) [ 4.074307] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.079604] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.084846] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.717780] device-mapper: uevent: version 1.0.3 [ 4.720414] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook..[ 5.467629] virtio_net virtio0 ens2: renamed from eth0 . [ 5.479380] random: fast init done [ 5.581701] scsi host0: ata_piix [ 5.674794] scsi host1: ata_piix [ 5.676544] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.678935] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.073336] dracut-initqueue[591]: RTNETLINK answers: File exists [ 10.348687] random: crng init done [ 10.350216] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.822977] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.021357] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.298766] SELinux: Disabled at runtime. [ 12.368906] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.379407] systemd[1]: Detected virtualization kvm. [ 12.381860] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.908962] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.912624] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.918714] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.922938] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.926278] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.935319] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.944265] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File S[ 12.972678] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ystems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.457215] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.841612] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.925530] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.009252] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.025702] EDAC sbridge: Ver: 1.1.2 [ 15.594508] Key type dns_resolver registered [ 15.917765] NFS: Registering the id_resolver key type [ 15.920143] Key type id_resolver registered [ 15.921967] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg330-server login: [ 41.946052] libcfs: loading out-of-tree module taints kernel. [ 41.967404] Key type ._llcrypt registered [ 41.968921] Key type .llcrypt registered [ 42.023269] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 50.788326] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 51.444249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 51.455645] alg: No test for adler32 (adler32-zlib) [ 52.509701] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 52.890164] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 54.512152] Key type lgssc registered [ 55.314891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 65.101150] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 87.109569] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 93.866679] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 93.882134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 95.023879] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 95.041327] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 95.110354] Lustre: lustre-MDT0000: new disk, initializing [ 95.169191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 95.182132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 97.504612] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 105.367807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 105.437459] Lustre: 6480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 105.458932] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 105.462385] Lustre: Skipped 1 previous similar message [ 105.521622] Lustre: lustre-MDT0001: new disk, initializing [ 105.567057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 105.586460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 105.592272] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 108.045741] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 111.297087] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 116.517829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 116.678331] Lustre: lustre-OST0000: new disk, initializing [ 116.680565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 116.683879] Lustre: 8418:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 116.713882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 118.397169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 118.406947] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 118.464034] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 120.203915] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 128.759335] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 128.821458] Lustre: lustre-OST0001: new disk, initializing [ 128.824807] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 128.828987] Lustre: 9492:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 128.866660] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 131.905767] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 137.732032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 137.738657] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 137.773356] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 139.485805] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 144.012869] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 147.152790] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing check_logdir /tmp/testlogs/ [ 149.633160] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing yml_node [ 151.950634] Lustre: DEBUG MARKER: Client: 2.17.57.82 [ 153.377115] Lustre: DEBUG MARKER: MDS: 2.17.57.82 [ 154.686430] Lustre: DEBUG MARKER: OSS: 2.17.57.82 [ 155.625813] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 1 09:29:17 EDT 2026 [ 166.003336] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 166.841223] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 182.240892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 182.246648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 182.255522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 183.778610] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 183.778626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 183.797684] Lustre: Skipped 3 previous similar messages [ 186.070802] Lustre: server umount lustre-MDT0000 complete [ 188.221207] LustreError: 6475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788269390 with bad export cookie 10770922879707984106 [ 188.221923] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 188.227576] LustreError: 6475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 188.903760] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 188.908106] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 188.917098] Lustre: Skipped 3 previous similar messages [ 188.925213] Lustre: Skipped 1 previous similar message [ 194.017214] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 194.017228] LustreError: 6490:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 194.020735] Lustre: Skipped 1 previous similar message [ 194.027421] LustreError: 6490:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 194.446419] Lustre: server umount lustre-MDT0001 complete [ 196.737889] Lustre: server umount lustre-OST0000 complete [ 198.961819] Lustre: server umount lustre-OST0001 complete [ 202.428622] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 207.230622] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 224.523429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 224.659574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 224.678642] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 224.734759] Lustre: lustre-MDT0000: new disk, initializing [ 224.784759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 224.794543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 227.163497] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 233.608422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 233.685791] Lustre: 15030:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 233.722977] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 233.726482] Lustre: Skipped 1 previous similar message [ 233.793852] Lustre: lustre-MDT0001: new disk, initializing [ 233.867914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 233.899312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 233.908060] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 236.303560] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 242.671387] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 246.293223] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 250.636195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 250.857720] Lustre: lustre-OST0000: new disk, initializing [ 250.860936] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 250.866586] Lustre: 16996:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 250.928878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 252.887929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 252.903683] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 252.966781] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 254.920401] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 260.919842] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 263.559649] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 263.674752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 266.344454] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 266.483513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 268.258142] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 268.261873] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 268.265511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 268.269185] Lustre: Skipped 1 previous similar message [ 268.276256] Lustre: Skipped 1 previous similar message [ 273.732790] Lustre: server umount lustre-OST0000 complete [ 278.500346] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 278.503630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 278.511970] Lustre: Skipped 1 previous similar message [ 278.525345] Lustre: Skipped 3 previous similar messages [ 290.272133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 290.354077] Lustre: server umount lustre-MDT0000 complete [ 292.190516] LustreError: 15022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788269494 with bad export cookie 10770922879707985639 [ 292.194501] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 292.198623] LustreError: 15022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 292.341684] Lustre: server umount lustre-MDT0001 complete [ 294.270781] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:31:36 (1788269496) [ 295.357818] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 296.171837] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:31:38 (1788269498) [ 299.449570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 299.701321] LustreError: 19402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 299.758940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 301.879263] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 305.122664] LustreError: 19403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 306.817234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 309.432174] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 313.333466] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 316.367598] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 320.573733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 325.198102] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 329.850487] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 340.448876] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 340.452684] LustreError: Skipped 1 previous similar message [ 340.454840] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.464046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 340.467521] Lustre: Skipped 4 previous similar messages [ 342.498565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 352.225213] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 352.335142] Lustre: server umount lustre-OST0000 complete [ 357.858290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 357.865824] Lustre: Skipped 1 previous similar message [ 361.418940] Lustre: server umount lustre-MDT0000 complete [ 362.977341] LustreError: 19403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 362.987676] LustreError: 19403:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 363.692712] LustreError: 19383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788269566 with bad export cookie 10770922879707986528 [ 363.697310] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 363.710298] LustreError: 19383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 363.894515] Lustre: server umount lustre-MDT0001 complete [ 370.025829] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 371.748702] Key type lgssc unregistered [ 371.942881] LNet: 22938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 371.948710] LNetError: 22938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 372.967770] LNet: Removed LNI 192.168.203.130@tcp [ 373.618372] Key type .llcrypt unregistered [ 373.622978] Key type ._llcrypt unregistered [ 387.434739] Key type ._llcrypt registered [ 387.437177] Key type .llcrypt registered [ 387.521874] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 388.141228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 388.152061] alg: No test for adler32 (adler32-zlib) [ 389.144442] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 389.303197] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 390.936217] Key type lgssc registered [ 391.716069] Lustre: Echo OBD driver; http://www.lustre.org/ [ 396.863379] loop: module loaded [ 396.916711] loop0: detected capacity change from 0 to 204800000 [ 397.062621] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 397.071182] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 397.313906] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 400.964365] loop0: detected capacity change from 0 to 204800000 [ 401.082639] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 401.090553] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 401.105641] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 404.836323] loop0: detected capacity change from 0 to 204800000 [ 408.690228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 408.706400] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 409.884295] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 409.934149] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 409.983694] Lustre: lustre-MDT0000: new disk, initializing [ 410.048130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 410.062277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 412.783848] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 418.745589] loop1: detected capacity change from 0 to 204800000 [ 422.380023] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 422.573578] Lustre: lustre-OST07c6: new disk, initializing [ 422.577065] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 422.580458] Lustre: Skipped 1 previous similar message [ 422.584377] Lustre: 26107:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 422.646345] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 424.438384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 424.447215] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 424.487882] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 426.568059] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 429.730139] Lustre: 25093:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 429.734295] Lustre: 25093:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 429.737557] Lustre: 25093:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 429.742167] Lustre: 25093:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 429.748282] Lustre: 25093:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 429.753378] Lustre: 25093:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 442.060148] Lustre: 25093:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 442.067110] Lustre: 25093:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 442.072368] Lustre: 25093:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 442.077070] Lustre: 25093:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 442.082486] Lustre: 25093:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 442.086887] Lustre: 25093:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 446.129677] Lustre: 25092:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 446.138902] Lustre: 25092:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 446.143383] Lustre: 25092:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 446.148819] Lustre: 25092:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 446.153768] Lustre: 25092:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 446.158254] Lustre: 25092:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 446.164247] Lustre: 25092:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 446.170575] Lustre: 25092:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 446.176217] Lustre: 25092:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 446.181185] Lustre: 25092:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 446.186465] Lustre: 25092:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 446.191809] Lustre: 25092:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 449.310361] Lustre: 25093:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 449.315776] Lustre: 25093:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 449.321038] Lustre: 25093:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 449.326721] Lustre: 25093:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 449.332065] Lustre: 25093:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 449.337235] Lustre: 25093:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 449.341960] Lustre: 25093:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 449.345030] Lustre: 25093:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 449.353543] Lustre: 25093:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 449.358699] Lustre: 25093:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 449.362792] Lustre: 25093:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 449.366759] Lustre: 25093:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 456.364445] Lustre: 25094:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 456.374363] Lustre: 25094:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 456.379186] Lustre: 25094:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 456.383652] Lustre: 25094:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 456.388768] Lustre: 25094:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 456.393719] Lustre: 25094:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 456.397656] Lustre: 25094:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 456.402959] Lustre: 25094:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 456.406935] Lustre: 25094:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 456.410886] Lustre: 25094:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 456.415888] Lustre: 25094:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 456.418863] Lustre: 25094:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 466.522398] Lustre: 25092:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 466.531429] Lustre: 25092:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 466.540533] Lustre: 25092:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 466.547963] Lustre: 25092:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 466.556902] Lustre: 25092:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 466.562444] Lustre: 25092:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 466.568422] Lustre: 25092:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 466.573815] Lustre: 25092:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 466.578690] Lustre: 25092:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 466.585265] Lustre: 25092:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 466.591816] Lustre: 25092:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 466.596879] Lustre: 25092:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 511.457839] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 511.463811] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 511.477032] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 515.304752] Lustre: server umount lustre-OST07c6 complete [ 519.818826] Lustre: server umount lustre-MDT0000 complete [ 527.245270] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 528.655620] Key type lgssc unregistered [ 528.885183] LNet: 28679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 528.888246] LNetError: 28679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 528.908940] LNet: Removed LNI 192.168.203.130@tcp [ 529.404253] Key type .llcrypt unregistered [ 529.407145] Key type ._llcrypt unregistered [ 539.922148] Key type ._llcrypt registered [ 539.924318] Key type .llcrypt registered [ 539.990420] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 540.511880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 540.530000] alg: No test for adler32 (adler32-zlib) [ 541.435044] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 541.567472] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 543.176817] Key type lgssc registered [ 543.660316] Lustre: Echo OBD driver; http://www.lustre.org/ [ 547.383746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 547.393257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.621780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 550.663270] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 555.128693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.353285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 557.852426] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 561.901177] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 564.593068] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 568.170267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 568.386029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 571.684717] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 573.543597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 575.374758] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 588.777722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 588.778046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.784000] LustreError: Skipped 1 previous similar message [ 588.794446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 589.497622] Lustre: server umount lustre-OST0000 complete [ 595.425680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 595.426920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 595.433541] Lustre: Skipped 1 previous similar message [ 595.441086] Lustre: Skipped 1 previous similar message [ 597.770359] Lustre: server umount lustre-MDT0000 complete [ 599.977807] LustreError: 29908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788269802 with bad export cookie 14587797315303303518 [ 599.981415] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 599.990032] LustreError: 29908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 610.373105] Lustre: server umount lustre-MDT0001 complete [ 616.143453] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 617.815564] Key type lgssc unregistered [ 617.994407] LNet: 33664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 618.003143] LNetError: 33664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 618.015833] LNet: Removed LNI 192.168.203.130@tcp [ 618.476533] Key type .llcrypt unregistered [ 618.478709] Key type ._llcrypt unregistered [ 629.233154] Key type ._llcrypt registered [ 629.234783] Key type .llcrypt registered [ 629.295484] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 629.750543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 629.799896] alg: No test for adler32 (adler32-zlib) [ 630.759900] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 630.916674] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 632.544342] Key type lgssc registered [ 633.194694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 637.279318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 637.290263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 638.523370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 640.480719] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 644.895485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 645.125681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 647.389559] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 651.234985] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 653.888576] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 657.279495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 657.445569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 660.401552] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 663.596932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 664.629247] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 674.788448] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 674.788482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 674.796220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 674.803218] Lustre: Skipped 1 previous similar message [ 676.080302] Lustre: server umount lustre-OST0000 complete [ 679.907138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 679.909929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 679.916194] Lustre: Skipped 2 previous similar messages [ 679.922275] Lustre: Skipped 1 previous similar message [ 684.293472] Lustre: server umount lustre-MDT0000 complete [ 685.027075] LustreError: 34913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 685.032709] LustreError: 34913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 686.402443] LustreError: 34893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788269889 with bad export cookie 3091047794115914490 [ 686.405115] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 686.410105] LustreError: 34893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 686.555882] Lustre: server umount lustre-MDT0001 complete [ 691.861347] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 693.374134] Key type lgssc unregistered [ 693.549954] LNet: 38446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.553567] LNetError: 38446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 693.562340] LNet: Removed LNI 192.168.203.130@tcp [ 693.998685] Key type .llcrypt unregistered [ 694.000756] Key type ._llcrypt unregistered [ 706.209482] Key type ._llcrypt registered [ 706.211611] Key type .llcrypt registered [ 706.273549] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 706.776688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 706.790132] alg: No test for adler32 (adler32-zlib) [ 707.710408] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 707.840404] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 709.456185] Key type lgssc registered [ 710.032395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 714.154259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 714.169052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 715.434163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 717.617928] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 721.817979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 722.004931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 724.022320] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 727.582158] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 730.225101] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 733.555229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 733.739227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 736.516518] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 739.961437] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 740.907939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 752.099716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 752.101496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 752.107094] Lustre: Skipped 2 previous similar messages [ 752.115412] Lustre: Skipped 2 previous similar messages [ 752.870194] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.30@tcp (stopping) [ 755.055185] Lustre: server umount lustre-MDT0000 complete [ 756.830375] LustreError: 39821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788269959 with bad export cookie 17306275836227527990 [ 756.830935] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 756.839391] LustreError: 39821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 757.217792] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 757.217826] LustreError: 39836:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 757.227511] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 757.235590] LustreError: 39836:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 757.975857] LustreError: 39835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 761.249142] LustreError: 39835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 761.249163] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 761.260861] Lustre: Skipped 1 previous similar message [ 763.029169] Lustre: server umount lustre-MDT0001 complete [ 781.676114] Lustre: server umount lustre-OST0000 complete [ 789.510623] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 790.922214] Key type lgssc unregistered [ 791.096591] LNet: 43700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.103634] LNetError: 43700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.114703] LNet: Removed LNI 192.168.203.130@tcp [ 791.525834] Key type .llcrypt unregistered [ 791.527425] Key type ._llcrypt unregistered [ 803.367174] Key type ._llcrypt registered [ 803.369176] Key type .llcrypt registered [ 803.431451] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 803.977685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 804.002699] alg: No test for adler32 (adler32-zlib) [ 804.917994] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 805.030490] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 806.640821] Key type lgssc registered [ 807.240623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 811.227121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 811.236923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 812.507093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 814.396188] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 818.865995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 819.062279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 820.971490] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 824.668531] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 827.368962] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 830.682484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 830.849281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 833.400952] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 836.944413] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 838.064904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 848.354777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 848.359807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 848.366683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 849.378822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 849.383896] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 849.385021] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 851.614620] Lustre: server umount lustre-OST0000 complete [ 865.251380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 865.259210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 865.267282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 865.270724] Lustre: Skipped 2 previous similar messages [ 869.856968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 869.857809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 871.332620] Lustre: server umount lustre-MDT0000 complete [ 873.069694] LustreError: 47672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788270075 with bad export cookie 13796897522781805490 [ 873.072605] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 873.075209] LustreError: 47672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 879.264872] Lustre: server umount lustre-MDT0001 complete [ 883.936451] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 885.347283] Key type lgssc unregistered [ 885.517830] LNet: 48801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.523651] LNetError: 48801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.536590] LNet: Removed LNI 192.168.203.130@tcp [ 886.013481] Key type .llcrypt unregistered [ 886.015743] Key type ._llcrypt unregistered [ 896.687300] Key type ._llcrypt registered [ 896.689080] Key type .llcrypt registered [ 896.744845] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 897.291842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 897.316436] alg: No test for adler32 (adler32-zlib) [ 898.200078] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 898.300705] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 899.888175] Key type lgssc registered [ 900.324261] Lustre: Echo OBD driver; http://www.lustre.org/ [ 903.576062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 903.585935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 904.788773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 906.632859] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 910.247145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 910.420947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 912.173987] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 915.175810] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 917.629767] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 920.582811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 920.673822] LustreError: 52014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 920.680402] LustreError: 52014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 920.764739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 923.183267] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 926.652923] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 930.282974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 934.146702] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 935.711261] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 946.549817] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-01 9h42m28s [ 947.220187] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 950.755734] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 950.757090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 950.762040] Lustre: Skipped 2 previous similar messages [ 950.765208] Lustre: Skipped 1 previous similar message [ 951.779587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 951.782321] Lustre: Skipped 3 previous similar messages [ 953.559226] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.30@tcp (stopping) [ 953.565525] Lustre: Skipped 1 previous similar message [ 954.221192] Lustre: server umount lustre-MDT0000 complete [ 955.571217] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 957.922543] LustreError: 50197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 957.930871] LustreError: 50197:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 958.271516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 958.335656] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 958.452302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 958.476996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 959.466305] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 960.141360] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 961.566438] Lustre: DEBUG MARKER: Wait for df (36622) ... [ 962.220597] Lustre: DEBUG MARKER: done [ 963.553642] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 963.556820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 963.562768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 964.578815] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 965.534899] Lustre: server umount lustre-OST0000 complete [ 969.134285] Lustre: server umount lustre-MDT0000 complete [ 969.697562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 969.698112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 969.699615] LustreError: 52880:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 969.716160] Lustre: Skipped 1 previous similar message [ 970.765570] LustreError: 50178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788270173 with bad export cookie 9378218101265942859 [ 970.769785] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 970.773812] LustreError: 50178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 972.926038] Lustre: server umount lustre-MDT0001 complete [ 976.993805] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 978.232428] Key type lgssc unregistered [ 978.399528] LNet: 55112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.403389] LNetError: 55112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.417976] LNet: Removed LNI 192.168.203.130@tcp [ 978.787232] Key type .llcrypt unregistered [ 978.789197] Key type ._llcrypt unregistered [ 995.145781] Key type ._llcrypt registered [ 995.147433] Key type .llcrypt registered [ 995.194412] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 995.584851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 995.638697] alg: No test for adler32 (adler32-zlib) [ 996.540270] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 996.645487] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 998.240157] Key type lgssc registered [ 998.731410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1002.141517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1002.156529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1003.284035] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1003.294055] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1003.360815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1005.012959] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1008.441850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1008.502081] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1008.518767] Lustre: 57299:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1008.629072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1010.248101] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1013.298869] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1015.508551] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1018.196235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1018.305075] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1018.391599] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1020.730345] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1021.397666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1023.602283] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1027.170634] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-01 9h43m49s [ 1027.783372] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1030.441913] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1031.378189] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1031.395753] LustreError: 57304:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1051.884052] Lustre: lustre-MDT0000: Client 8713b952-3818-4cae-990d-ddbdcf96f787 (at 192.168.203.30@tcp) reconnecting [ 1071.440078] LustreError: 57304:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1072.266133] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1073.157894] Lustre: DEBUG MARKER: done [ 1076.708313] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1076.712123] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1076.719667] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1077.731230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1077.731239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1077.734031] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1081.058675] Lustre: server umount lustre-OST0000 complete [ 1082.850171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1082.851305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1082.856420] Lustre: Skipped 1 previous similar message [ 1082.859489] Lustre: Skipped 1 previous similar message [ 1088.736500] Lustre: server umount lustre-MDT0000 complete [ 1090.137531] LustreError: 58039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788270292 with bad export cookie 18408277526202372685 [ 1090.140429] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1090.143076] LustreError: 58039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1090.263265] Lustre: server umount lustre-MDT0001 complete [ 1094.138551] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 1095.223603] Key type lgssc unregistered [ 1095.357558] LNet: 61623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1095.361408] LNetError: 61623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1095.375413] LNet: Removed LNI 192.168.203.130@tcp [ 1095.682239] Key type .llcrypt unregistered [ 1095.684061] Key type ._llcrypt unregistered [ 1118.730479] Key type ._llcrypt registered [ 1118.732291] Key type .llcrypt registered [ 1118.824691] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1120.049947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1120.215061] alg: No test for adler32 (adler32-zlib) [ 1121.153156] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 1121.320068] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 1122.936895] Key type lgssc registered [ 1124.010389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1127.539748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1127.553381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1128.721503] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1128.746853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1128.899373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1129.790302] Lustre: Failing over lustre-MDT0000 [ 1132.144729] Lustre: server umount lustre-MDT0000 complete [ 1138.030407] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:45:39 (1788270339) [ 1143.703571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1144.099151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1148.565761] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1157.639478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1157.794468] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1157.866991] Lustre: 64397:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1158.455884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1165.463200] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1173.335856] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1179.724868] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1185.104649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1185.275904] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1185.431493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1189.399721] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1194.627575] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1195.593154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1203.982439] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1209.825258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1209.831815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1209.845151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1210.851282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1210.852842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1210.867646] Lustre: Skipped 2 previous similar messages [ 1213.624747] Lustre: server umount lustre-MDT0000 complete [ 1215.847844] LustreError: 64392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788270418 with bad export cookie 3701449860731155525 [ 1215.852342] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1215.859133] LustreError: 64392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1215.971873] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.982955] Lustre: Skipped 1 previous similar message [ 1216.093601] Lustre: server umount lustre-MDT0001 complete [ 1218.179506] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1220.990441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1226.079080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1226.208771] LustreError: 63321:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f5b4a37d x1875137342248192/t0(0) o250->MGC192.168.203.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1226.429994] LustreError: 66248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1226.449033] LustreError: 66248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1226.500574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1226.579174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1229.197676] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1231.842069] LustreError: 66249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1234.483706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1234.771859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1235.826433] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1237.578664] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1241.003679] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1242.254537] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1246.011663] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1250.274118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1250.280217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1250.292085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1256.399972] Lustre: server umount lustre-MDT0000 complete [ 1256.420316] LustreError: 68167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1256.433039] LustreError: 68167:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1258.471836] LustreError: 66255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788270461 with bad export cookie 3701449860731157807 [ 1258.478764] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1258.480354] LustreError: 66255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1258.671561] Lustre: server umount lustre-MDT0001 complete [ 1262.455218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1268.309438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1268.705605] LustreError: 63321:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e9fc5188 x1875137342270720/t0(0) o250->MGC192.168.203.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1268.934544] LustreError: 66249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1268.992705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1269.036964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1271.497700] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1274.340412] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1274.340911] LustreError: 68899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1274.348573] Lustre: Skipped 2 previous similar messages [ 1274.370556] LustreError: 68899:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1277.270445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1277.386072] Lustre: 63325:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788270464/real 1788270464] req@00000000ff3db37c x1875137342270336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788270480 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1279.459203] Lustre: 63322:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788270466/real 1788270466] req@00000000192fc403 x1875137342270464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788270482 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1280.348922] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1281.643143] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1283.556465] Lustre: 63323:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788270469/real 1788270469] req@00000000d7387969 x1875137342270592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788270485 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1285.215113] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1288.617696] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1292.260119] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1292.264575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1292.269179] Lustre: Skipped 1 previous similar message [ 1292.282322] Lustre: Skipped 5 previous similar messages [ 1297.380361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1297.390996] Lustre: Skipped 2 previous similar messages [ 1298.356494] Lustre: server umount lustre-MDT0000 complete [ 1300.779837] LustreError: 64391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788270503 with bad export cookie 3701449860731159186 [ 1300.786359] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1300.789634] LustreError: 64391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1300.973821] Lustre: server umount lustre-MDT0001 complete [ 1305.400698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1308.209871] Lustre: DEBUG MARKER: files compared the same [ 1320.179795] Lustre: server umount lustre-OST0000 complete [ 1331.121939] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 1333.065456] Key type lgssc unregistered [ 1333.284205] LNet: 74118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1333.289590] LNetError: 74118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1333.311526] LNet: Removed LNI 192.168.203.130@tcp [ 1333.907256] Key type .llcrypt unregistered [ 1333.909194] Key type ._llcrypt unregistered [ 1349.974440] Key type ._llcrypt registered [ 1349.976339] Key type .llcrypt registered [ 1350.073487] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1350.728733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1350.755489] alg: No test for adler32 (adler32-zlib) [ 1351.726375] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 1351.976212] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 1353.640152] Key type lgssc registered [ 1354.368893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1359.976273] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1359.986522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1361.261207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1364.315439] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug +malloc all [ 1370.592805] LustreError: 75513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1370.612785] LustreError: 75513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1370.679248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1370.994854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1374.306485] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug +malloc all [ 1380.174308] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1384.432396] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1388.924508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1389.126924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1394.078243] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug +malloc all [ 1399.385457] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1405.546891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1412.067707] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1412.071437] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1412.076260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1416.181450] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1416.189711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1416.199547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1417.840431] Lustre: server umount lustre-OST0000 complete [ 1422.305096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1422.310670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1422.322829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1422.330040] Lustre: Skipped 1 previous similar message [ 1426.407527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1426.408176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1431.521685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1431.529157] Lustre: Skipped 2 previous similar messages [ 1436.128897] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1436.216601] Lustre: server umount lustre-MDT0000 complete [ 1436.644815] LustreError: 75514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1436.660114] LustreError: 75514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1439.285205] LustreError: 75494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788270641 with bad export cookie 3799155253924383548 [ 1439.296238] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1439.305610] LustreError: 75494:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1439.541949] Lustre: server umount lustre-MDT0001 complete [ 1448.122610] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 1450.178955] Key type lgssc unregistered [ 1450.421698] LNet: 79071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1450.429341] LNetError: 79071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1450.447839] LNet: Removed LNI 192.168.203.130@tcp [ 1450.961907] Key type .llcrypt unregistered [ 1450.964256] Key type ._llcrypt unregistered [ 1466.977440] Key type ._llcrypt registered [ 1466.980264] Key type .llcrypt registered [ 1467.051677] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1467.636045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1467.728568] alg: No test for adler32 (adler32-zlib) [ 1468.816801] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 1468.998017] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 1470.625293] Key type lgssc registered [ 1471.503888] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1477.264642] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1477.292132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1492.832374] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1497.953428] LustreError: 80440:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.130@tcp: failed processing log, type 4: rc = -110 [ 1524.576216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1528.640568] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1534.297626] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1539.569373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1539.668072] Lustre: *** cfs_fail_loc=706, val=0*** [ 1539.942050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1540.004988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1543.051268] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1546.212701] LustreError: 81525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1546.254815] LustreError: 81525:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 1549.641084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1549.880848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1552.554207] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1557.193373] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1561.496632] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1564.134684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1564.138297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1564.147448] Lustre: Skipped 1 previous similar message [ 1564.157263] Lustre: Skipped 1 previous similar message [ 1569.255608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1569.260038] Lustre: Skipped 1 previous similar message [ 1569.363249] Lustre: server umount lustre-OST0000 complete [ 1574.371607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1574.375621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1574.379446] Lustre: Skipped 1 previous similar message [ 1578.525489] Lustre: server umount lustre-MDT0000 complete [ 1579.489208] LustreError: 81529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1579.504427] LustreError: 81529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1581.229033] LustreError: 80447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788270783 with bad export cookie 16550857481182049283 [ 1581.245109] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1581.247599] LustreError: 80447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1581.632613] Lustre: server umount lustre-MDT0001 complete [ 1589.095150] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 1591.231680] Key type lgssc unregistered [ 1591.442576] LNet: 84057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1591.449340] LNetError: 84057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1591.464507] LNet: Removed LNI 192.168.203.130@tcp [ 1592.103663] Key type .llcrypt unregistered [ 1592.106057] Key type ._llcrypt unregistered [ 1608.095806] Key type ._llcrypt registered [ 1608.098355] Key type .llcrypt registered [ 1608.172241] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1608.969938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1608.996310] alg: No test for adler32 (adler32-zlib) [ 1609.993638] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 1610.189329] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 1611.808505] Key type lgssc registered [ 1612.725639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1618.285568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1618.301764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1622.546748] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1627.080769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1627.410600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1630.107862] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1632.742846] LustreError: 86137:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1632.764643] LustreError: 86137:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1635.322675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1635.544395] LustreError: 86140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1635.557310] LustreError: 86140:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1635.570798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1637.601187] LustreError: 86141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1637.617151] LustreError: 86141:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1639.049678] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1642.980107] LustreError: 86135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1642.995601] LustreError: 86135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1644.264200] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1644.272294] Lustre: MGS: Client faa2810a-9ff3-4653-822b-84389b2f6a93 (at 0@lo) reconnecting [ 1644.292427] LustreError: 86151:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1644.433255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1644.494402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1647.120223] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1659.876564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1659.879047] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1659.883324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1659.891854] Lustre: Skipped 1 previous similar message [ 1663.617859] Lustre: server umount lustre-OST0000 complete [ 1670.114428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1670.114935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1670.115720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1670.140961] Lustre: Skipped 2 previous similar messages [ 1671.857520] Lustre: server umount lustre-MDT0000 complete [ 1674.335275] Lustre: server umount lustre-MDT0001 complete [ 1676.627643] Lustre: server umount MGS complete [ 1683.170978] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:54:45 (1788270885) [ 1697.192839] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 1701.633233] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1717.935472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1719.769525] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1724.720489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1724.837488] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1724.862303] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1724.905155] Lustre: lustre-MDT0001: new disk, initializing [ 1724.944952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1724.957727] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1724.967970] Lustre: 93298:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1724.974936] LustreError: 93298:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1726.974749] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1732.738513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1732.877302] Lustre: lustre-OST0000: new disk, initializing [ 1732.879630] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1732.883681] Lustre: 94063:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1732.933559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1734.242383] Lustre: 94097:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000d7ab0587 x1875137855000832/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788270952 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1735.848581] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1740.932959] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1744.166406] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1744.174585] Lustre: MGS: Client 8ecff784-0b8b-4140-b30c-7865b813d5ab (at 0@lo) reconnecting [ 1744.198781] Lustre: 95109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1744.207720] LustreError: 93284:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1744.213434] LustreError: 93284:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1744.218936] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1744.263903] Lustre: lustre-MDT0000: new disk, initializing [ 1744.331184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1744.349505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1744.406704] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1745.382098] Lustre: 95134:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000004b2a3147 x1875137855007360/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788270964 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1746.588368] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1754.477624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1754.487780] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1754.550922] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1780.197601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.199317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1780.199417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1780.203479] Lustre: Skipped 1 previous similar message [ 1780.206357] Lustre: Skipped 1 previous similar message [ 1782.701996] Lustre: server umount lustre-OST0000 complete [ 1785.314714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1785.315455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1785.321323] Lustre: Skipped 1 previous similar message [ 1785.328729] Lustre: Skipped 2 previous similar messages [ 1790.433953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1790.436823] Lustre: Skipped 1 previous similar message [ 1790.480775] Lustre: server umount lustre-MDT0000 complete [ 1806.304100] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1806.405388] Lustre: server umount lustre-MDT0001 complete [ 1807.925085] Lustre: server umount MGS complete [ 1812.353688] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:56:54 (1788271014) [ 1819.219850] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 1820.341496] Key type lgssc unregistered [ 1820.474537] LNet: 97634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1820.478301] LNetError: 97634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1820.491405] LNet: Removed LNI 192.168.203.130@tcp [ 1820.831330] Key type .llcrypt unregistered [ 1820.832893] Key type ._llcrypt unregistered [ 1828.544191] Key type ._llcrypt registered [ 1828.545808] Key type .llcrypt registered [ 1828.586033] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1828.957360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1829.015946] alg: No test for adler32 (adler32-zlib) [ 1829.870265] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 1829.958541] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 1831.536160] Key type lgssc registered [ 1831.912309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1834.544894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1834.551826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1835.695910] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1837.092742] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1840.115257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1840.250556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1841.671886] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1844.233647] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1846.194194] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1848.671506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1848.811525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1850.863843] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1853.599156] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1862.123404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1867.234625] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1867.237747] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1867.238892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1867.242730] Lustre: Skipped 1 previous similar message [ 1867.248075] Lustre: Skipped 1 previous similar message [ 1868.567623] Lustre: server umount lustre-OST0000 complete [ 1871.328789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1871.333145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1871.340675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1871.893488] Lustre: server umount lustre-MDT0000 complete [ 1872.354392] LustreError: 98832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1872.354452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1872.361040] LustreError: 98832:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1873.078312] LustreError: 98812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788271075 with bad export cookie 16994984766457279475 [ 1873.081036] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1873.082431] LustreError: 98812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1873.190303] Lustre: server umount lustre-MDT0001 complete [ 1876.652539] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 1877.685208] Key type lgssc unregistered [ 1877.810395] LNet: 102385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1877.812929] LNetError: 102385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1877.825343] LNet: Removed LNI 192.168.203.130@tcp [ 1878.134136] Key type .llcrypt unregistered [ 1878.135745] Key type ._llcrypt unregistered [ 1884.686480] Key type ._llcrypt registered [ 1884.688066] Key type .llcrypt registered [ 1884.732833] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1885.063905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1885.091547] alg: No test for adler32 (adler32-zlib) [ 1885.957602] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 1886.054674] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 1887.640230] Key type lgssc registered [ 1888.089536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1890.331423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1890.331441] LustreError: 103450:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 1890.338306] Lustre: Skipped 1 previous similar message [ 1890.340045] LustreError: 103451:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 1890.343203] LustreError: 103450:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4988 [ 1890.346842] LustreError: 103450:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1890.351332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1890.352704] LustreError: 103450:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1891.513084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1894.234732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1894.339303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1895.813957] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1897.544452] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing lsmod [ 1898.552698] LustreError: 104655:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 1898.552830] LustreError: 104656:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 1898.557978] LustreError: 104655:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4998 [ 1898.561113] LustreError: 104655:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1898.563890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1898.567644] LustreError: 104655:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1898.690401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1903.715250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1903.716814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1903.722689] Lustre: Skipped 1 previous similar message [ 1905.753087] Lustre: server umount lustre-MDT0000 complete [ 1907.045264] LustreError: 104842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788271109 with bad export cookie 4204058139936554762 [ 1907.048318] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1907.050892] LustreError: 104842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1907.808342] LustreError: 103919:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000042bcef29 x1875138144386432/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1907.859065] Lustre: server umount lustre-MDT0001 complete [ 1918.428204] Lustre: server umount lustre-OST0000 complete [ 1921.328353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1921.477318] LustreError: 105718:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1921.485748] LustreError: 105718:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1921.501768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1922.856356] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1925.741522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1925.871986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1927.199986] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1929.601734] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1931.497847] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1933.706170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1935.661259] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1937.993949] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1947.110928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1951.200690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1951.203914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1951.207926] Lustre: Skipped 1 previous similar message [ 1951.209842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1951.211940] Lustre: Skipped 2 previous similar messages [ 1952.226386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1952.227442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1952.227781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1952.238561] Lustre: Skipped 1 previous similar message [ 1955.853941] Lustre: server umount lustre-OST0000 complete [ 1957.345706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1957.346300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1957.351569] Lustre: Skipped 1 previous similar message [ 1962.465651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1962.468256] Lustre: Skipped 2 previous similar messages [ 1963.158411] Lustre: server umount lustre-MDT0000 complete [ 1964.379564] LustreError: 105699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788271167 with bad export cookie 4204058139936555448 [ 1964.382623] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1964.384405] LustreError: 105699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1964.503205] Lustre: server umount lustre-MDT0001 complete [ 1968.228809] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 1969.352760] Key type lgssc unregistered [ 1969.497338] LNet: 109269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1969.500653] LNetError: 109269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1969.508341] LNet: Removed LNI 192.168.203.130@tcp [ 1969.830441] Key type .llcrypt unregistered [ 1969.832077] Key type ._llcrypt unregistered [ 1979.675914] Key type ._llcrypt registered [ 1979.677480] Key type .llcrypt registered [ 1979.721556] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1980.083707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1980.098678] alg: No test for adler32 (adler32-zlib) [ 1980.964984] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 1981.063970] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 1982.648153] Key type lgssc registered [ 1983.081088] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1986.047191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1986.053961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1987.214125] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1988.654856] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1991.615239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1991.744953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1993.178155] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1995.706087] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1997.646580] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2000.034857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2000.159910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2002.156181] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2004.533698] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2012.454067] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 2016.862512] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2017.583934] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2022.882020] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2022.882249] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2022.884950] LustreError: Skipped 1 previous similar message [ 2022.889197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2022.893682] Lustre: Skipped 1 previous similar message [ 2024.652068] Lustre: server umount lustre-OST0000 complete [ 2027.488760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2027.491753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2027.496109] Lustre: Skipped 1 previous similar message [ 2027.498585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2033.120928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2033.122780] Lustre: Skipped 2 previous similar messages [ 2038.240971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2038.243012] Lustre: Skipped 1 previous similar message [ 2040.288118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2040.351020] Lustre: server umount lustre-MDT0000 complete [ 2041.531295] LustreError: 110644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788271244 with bad export cookie 5727841240867071658 [ 2041.534932] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2041.536593] LustreError: 110644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2041.644214] Lustre: server umount lustre-MDT0001 complete [ 2045.216561] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 2046.290411] Key type lgssc unregistered [ 2046.419525] LNet: 114321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2046.422790] LNetError: 114321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2046.431459] LNet: Removed LNI 192.168.203.130@tcp [ 2046.744369] Key type .llcrypt unregistered [ 2046.746056] Key type ._llcrypt unregistered [ 2053.698476] Key type ._llcrypt registered [ 2053.699924] Key type .llcrypt registered [ 2053.740681] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2054.113328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2054.143389] alg: No test for adler32 (adler32-zlib) [ 2055.011515] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 2055.106882] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 2056.688166] Key type lgssc registered [ 2057.084589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2059.893434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2059.900179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2061.057292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2062.453331] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2065.471178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.604570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2067.054235] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2069.460515] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2071.294380] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2073.608349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2073.723068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2075.601584] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2078.072854] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2085.990700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2091.490501] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2091.491449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2091.492801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2091.494749] Lustre: Skipped 1 previous similar message [ 2096.612888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2096.615348] Lustre: Skipped 2 previous similar messages [ 2097.360090] Lustre: server umount lustre-OST0000 complete [ 2101.730703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2101.731621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2101.733573] LustreError: Skipped 1 previous similar message [ 2101.734295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2101.737775] Lustre: Skipped 1 previous similar message [ 2104.723137] Lustre: server umount lustre-MDT0000 complete [ 2105.937126] LustreError: 115527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788271308 with bad export cookie 9382173672131022374 [ 2105.939858] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2105.942045] LustreError: 115527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2106.047412] Lustre: server umount lustre-MDT0001 complete [ 2109.400280] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 2110.413859] Key type lgssc unregistered [ 2110.531306] LNet: 119099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2110.533751] LNetError: 119099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2110.542282] LNet: Removed LNI 192.168.203.130@tcp [ 2110.852290] Key type .llcrypt unregistered [ 2110.853831] Key type ._llcrypt unregistered [ 2119.202137] Key type ._llcrypt registered [ 2119.203625] Key type .llcrypt registered [ 2119.238545] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2119.563159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2119.602805] alg: No test for adler32 (adler32-zlib) [ 2120.460903] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 2120.545037] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 2122.128199] Key type lgssc registered [ 2122.463368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2124.983269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2124.989464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2126.126736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2127.390242] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2130.119286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.245277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2131.504844] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2133.827188] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2135.625634] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2137.801021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2137.912336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2139.675822] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2141.985126] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2151.212544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2154.960337] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2156.049227] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2156.051519] Lustre: Skipped 2 previous similar messages [ 2157.283505] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2157.285840] Lustre: Skipped 2 previous similar messages [ 2160.631727] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2160.639972] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2164.286487] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.30@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2164.290285] Lustre: Skipped 2 previous similar messages [ 2170.871904] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.30@tcp 192.168.0.[2,10]@tcp [ 2172.060222] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.30/24@tcp in log lustre-MDT0000 [ 2172.062959] Lustre: Skipped 2 previous similar messages [ 2179.577340] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.30/24@tcp [ 2179.579902] Lustre: Skipped 1 previous similar message [ 2182.114456] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2182.114595] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2182.120995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2182.126739] Lustre: Skipped 1 previous similar message [ 2187.221295] Lustre: server umount lustre-OST0000 complete [ 2191.840879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2191.843903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2191.848260] Lustre: Skipped 1 previous similar message [ 2191.850964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2194.585955] Lustre: server umount lustre-MDT0000 complete [ 2195.892242] LustreError: 123143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788271398 with bad export cookie 16067322612526080103 [ 2195.896967] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2195.899526] LustreError: 123143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2196.026126] Lustre: server umount lustre-MDT0001 complete [ 2199.666212] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 2200.676577] Key type lgssc unregistered [ 2200.798382] LNet: 125735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2200.801320] LNetError: 125735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2200.810409] LNet: Removed LNI 192.168.203.130@tcp [ 2201.127867] Key type .llcrypt unregistered [ 2201.129250] Key type ._llcrypt unregistered [ 2210.888292] Key type ._llcrypt registered [ 2210.890116] Key type .llcrypt registered [ 2210.929437] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2211.240947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2211.265717] alg: No test for adler32 (adler32-zlib) [ 2212.124269] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 2212.214823] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 2213.800169] Key type lgssc registered [ 2214.190515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2216.669774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2216.675440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2217.830879] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2217.833995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2217.840741] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.30/24@tcp [ 2219.156503] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2221.762884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2221.856136] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2221.862535] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.30/24@tcp [ 2221.875995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2223.092632] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2225.624369] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2227.711803] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2231.185217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2231.383407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2232.574585] hrtimer: interrupt took 5634188 ns [ 2236.934045] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2242.687610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2245.619359] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2253.281399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2253.284943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2253.293569] LustreError: Skipped 1 previous similar message [ 2253.305304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2258.402608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2258.415585] Lustre: Skipped 2 previous similar messages [ 2259.207607] Lustre: server umount lustre-OST0000 complete [ 2263.009740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2263.020320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2263.046565] Lustre: Skipped 1 previous similar message [ 2263.094958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2268.515719] Lustre: server umount lustre-MDT0000 complete [ 2268.643045] LustreError: 128708:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2268.655045] LustreError: 128708:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2271.279823] LustreError: 127254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788271473 with bad export cookie 1381584075795260423 [ 2271.291423] LustreError: 127254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2271.293145] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2271.523872] Lustre: server umount lustre-MDT0001 complete [ 2277.649965] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 2279.131513] Key type lgssc unregistered [ 2279.290053] LNet: 130857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2279.295570] LNetError: 130857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2279.306583] LNet: Removed LNI 192.168.203.130@tcp [ 2279.703282] Key type .llcrypt unregistered [ 2279.705055] Key type ._llcrypt unregistered [ 2291.838284] Key type ._llcrypt registered [ 2291.840322] Key type .llcrypt registered [ 2291.902843] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 2299.680654] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2300.069034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2300.078282] alg: No test for adler32 (adler32-zlib) [ 2300.996357] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 2301.108916] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 2302.704103] Key type lgssc registered [ 2303.164220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2322.840438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2322.850425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2324.057758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2324.075782] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2324.141991] Lustre: lustre-MDT0000: new disk, initializing [ 2324.208949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2324.220838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2328.658315] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2340.430599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2340.624431] Lustre: 134232:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2340.717293] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2340.721560] Lustre: Skipped 1 previous similar message [ 2340.834980] Lustre: lustre-MDT0001: new disk, initializing [ 2340.954975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2341.005935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2341.028951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2347.027188] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2357.032713] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2362.869195] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2370.689044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2370.968332] Lustre: lustre-OST0000: new disk, initializing [ 2370.970989] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2370.979100] Lustre: 136201:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2371.030857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2376.279926] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2380.817415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2380.830311] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2380.939113] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2383.740591] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2387.950917] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2388.285861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2392.903139] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2393.175598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2396.144882] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2396.148419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2396.183120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2396.203595] Lustre: Skipped 1 previous similar message [ 2401.261049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2401.274402] Lustre: Skipped 1 previous similar message [ 2406.370697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2406.383155] Lustre: Skipped 1 previous similar message [ 2409.954461] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2410.150067] Lustre: server umount lustre-OST0000 complete [ 2416.609746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2416.626470] Lustre: Skipped 1 previous similar message [ 2416.629324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2416.640159] Lustre: Skipped 1 previous similar message [ 2421.730309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2421.738317] Lustre: Skipped 1 previous similar message [ 2427.362843] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2427.541229] Lustre: server umount lustre-MDT0000 complete [ 2430.130399] LustreError: 134223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788271632 with bad export cookie 10317545557254809697 [ 2430.144422] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2430.151946] LustreError: 134223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2430.434092] Lustre: server umount lustre-MDT0001 complete [ 2436.209355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2436.652914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2440.117977] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2441.699125] LustreError: 138420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2441.711782] LustreError: 138420:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2446.761666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2446.817452] LustreError: 138421:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.054232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2450.415137] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2456.527159] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2461.150891] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2466.336148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2466.593719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2470.732237] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2476.610779] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2480.365971] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2482.625648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2486.242855] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2486.382718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2492.935246] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2493.097880] Lustre: lustre-OST0001: new disk, initializing [ 2493.111786] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2493.125369] Lustre: 141803:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2493.209976] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2497.772808] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2500.691924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2500.708674] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2500.791859] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2505.173859] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2505.435126] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2508.883661] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2509.093813] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2513.331781] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ca2887d7800.ost_server_uuid 50 [ 2514.654386] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8ca2887d7800.ost_server_uuid in FULL state after 0 sec [ 2521.570590] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2521.576770] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2521.578634] LustreError: Skipped 1 previous similar message [ 2521.589026] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2521.598455] Lustre: Skipped 3 previous similar messages [ 2526.971322] Lustre: server umount lustre-OST0001 complete [ 2531.814372] LustreError: 140242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2531.820196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2531.840440] LustreError: 140242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2531.856623] Lustre: Skipped 2 previous similar messages [ 2536.937274] LustreError: 140241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2536.971815] LustreError: 140241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2542.054064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2542.056805] LustreError: 141457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2542.066670] Lustre: Skipped 7 previous similar messages [ 2542.085886] LustreError: 141457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2544.096563] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2544.358779] Lustre: server umount lustre-OST0000 complete [ 2549.732616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2549.753725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2554.529249] Lustre: server umount lustre-MDT0000 complete [ 2557.410857] LustreError: 138416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2557.429827] LustreError: 138416:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2557.722420] LustreError: 138402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788271760 with bad export cookie 10317545557254810572 [ 2557.733681] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2558.067795] Lustre: server umount lustre-MDT0001 complete [ 2576.354207] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 2579.605613] Key type lgssc unregistered [ 2580.043963] LNet: 144807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2580.062667] LNetError: 144807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2580.078356] LNet: Removed LNI 192.168.203.130@tcp [ 2580.910835] Key type .llcrypt unregistered [ 2580.912887] Key type ._llcrypt unregistered [ 2619.696270] Key type ._llcrypt registered [ 2619.702113] Key type .llcrypt registered [ 2619.872163] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 2633.491484] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2634.179942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2634.200637] alg: No test for adler32 (adler32-zlib) [ 2635.354703] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 2635.629666] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 2637.297375] Key type lgssc registered [ 2638.263967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2669.578674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2669.600067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2670.856343] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2670.877868] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2670.950304] Lustre: lustre-MDT0000: new disk, initializing [ 2671.045122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2671.068491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2674.708452] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2683.776857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2683.884965] Lustre: 148961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2683.938595] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2683.943916] Lustre: Skipped 1 previous similar message [ 2684.050396] Lustre: lustre-MDT0001: new disk, initializing [ 2684.189490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2684.236796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2684.260843] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2688.016972] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2696.213164] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2700.835250] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2706.820450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2707.094282] Lustre: lustre-OST0000: new disk, initializing [ 2707.101747] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2707.113365] Lustre: 150929:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2707.207982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2712.255630] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2714.191305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2714.203651] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2714.323618] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2720.499546] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2723.985873] Lustre: Failing over lustre-OST0000 [ 2724.118909] Lustre: server umount lustre-OST0000 complete [ 2741.223913] Lustre: 147183:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788271927/real 1788271927] req@00000000b9fc29c7 x1875138930834176/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788271943 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2741.232984] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.248021] Lustre: 147183:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2741.271593] Lustre: Skipped 1 previous similar message [ 2742.634954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2742.840577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2742.852572] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2743.572860] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2743.843675] Lustre: 147183:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788271929/real 1788271929] req@00000000a2f2e617 x1875138930835328/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788271945 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2745.824564] Lustre: 147186:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788271932/real 1788271932] req@000000001930dcf7 x1875138930836096/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788271948 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2748.915364] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2750.994231] Lustre: 147184:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788271937/real 1788271937] req@00000000e26da890 x1875138930837248/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788271953 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2751.023601] Lustre: 147184:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2751.071342] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2751.072995] Lustre: lustre-OST0000: Recovery over after 0:08, of 3 clients 3 recovered and 0 were evicted. [ 2752.495594] Lustre: Failing over lustre-MDT0000 [ 2752.829775] Lustre: server umount lustre-MDT0000 complete [ 2755.552148] Lustre: 147183:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788271942/real 1788271942] req@0000000029da0df6 x1875138930838400/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788271958 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2755.576892] Lustre: 147183:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2756.067250] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2756.088940] Lustre: Skipped 1 previous similar message [ 2756.093992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2758.891173] LustreError: 148967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2758.915418] LustreError: 148967:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2761.187876] LustreError: 148971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2761.221827] LustreError: 148971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2763.996195] LustreError: 148966:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.305108] LustreError: 148971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2766.320918] LustreError: 148971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2771.430992] LustreError: 148972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2771.441265] LustreError: 148972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2771.755547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2771.886900] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2772.154343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2772.206863] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2774.238866] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2776.150947] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2777.585677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2777.595255] Lustre: Skipped 1 previous similar message [ 2777.623439] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2777.693308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2787.297372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2787.304624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2787.316028] Lustre: Skipped 1 previous similar message [ 2787.325580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2792.935932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2792.951308] Lustre: Skipped 2 previous similar messages [ 2798.049335] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2798.056218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2798.066203] Lustre: Skipped 1 previous similar message [ 2798.291125] Lustre: server umount lustre-OST0000 complete [ 2803.169437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2803.170302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2803.179629] Lustre: Skipped 1 previous similar message [ 2803.180357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2807.688787] Lustre: server umount lustre-MDT0000 complete [ 2808.293398] LustreError: 148967:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2808.324434] LustreError: 148967:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2811.311933] LustreError: 148952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788272013 with bad export cookie 7355428232044239477 [ 2811.317234] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2811.319365] LustreError: 148952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2811.646614] Lustre: server umount lustre-MDT0001 complete [ 2821.021702] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 2823.296634] Key type lgssc unregistered [ 2823.700803] LNet: 154585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2823.708025] LNetError: 154585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2824.756040] LNet: Removed LNI 192.168.203.130@tcp [ 2825.495474] Key type .llcrypt unregistered [ 2825.504558] Key type ._llcrypt unregistered [ 2857.115609] Key type ._llcrypt registered [ 2857.118359] Key type .llcrypt registered [ 2857.196850] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2858.037199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2858.066616] alg: No test for adler32 (adler32-zlib) [ 2859.052409] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 2859.271456] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 2860.928700] Key type lgssc registered [ 2861.838411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2867.811991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2867.832594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2869.238973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2872.492680] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2878.433668] LustreError: 156115:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2878.450520] LustreError: 156115:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2879.881857] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2883.307547] LustreError: 156144:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2883.313516] LustreError: 156144:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2883.318572] LustreError: 156144:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 2883.553216] Lustre: server umount lustre-MDT0000 complete [ 2895.873531] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 2903.565950] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2931.217576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2931.478103] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2931.511337] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2931.611724] Lustre: lustre-MDT0000: new disk, initializing [ 2931.666165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2931.682574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2935.439369] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2944.813511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2944.922634] Lustre: 159445:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2944.971672] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2944.975088] Lustre: Skipped 1 previous similar message [ 2945.048043] Lustre: lustre-MDT0001: new disk, initializing [ 2945.153405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2945.209279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2945.219844] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2948.734818] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2956.215964] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2960.200757] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2965.033403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2965.218989] Lustre: lustre-OST0000: new disk, initializing [ 2965.222364] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2965.228744] Lustre: 161416:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2965.296267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2969.196684] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2973.253962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2973.269647] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2973.379837] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2976.087593] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2979.040033] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2980.344745] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:16:22 (1788272182) [ 3193.825852] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3193.830106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3193.839803] Lustre: Skipped 2 previous similar messages [ 3193.850586] Lustre: Skipped 2 previous similar messages [ 3197.320647] Lustre: server umount lustre-MDT0000 complete [ 3198.946503] LustreError: 159458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3198.972670] LustreError: 159458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3201.183091] LustreError: 159439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788272403 with bad export cookie 10976961200842221288 [ 3201.183499] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3201.203306] LustreError: 159439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3204.070429] LustreError: 159453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3204.088696] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3204.120042] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3206.112698] LustreError: 163189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3206.113051] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3207.615897] Lustre: server umount lustre-MDT0001 complete [ 3217.982112] Lustre: server umount lustre-OST0000 complete [ 3228.788927] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 3238.581183] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 3274.168826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3274.375994] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3274.407342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3274.526637] Lustre: lustre-MDT0000: new disk, initializing [ 3274.630723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3274.647270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3279.420331] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3290.317949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3290.447718] Lustre: 166073:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3290.512517] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3290.519383] Lustre: Skipped 1 previous similar message [ 3290.628595] Lustre: lustre-MDT0001: new disk, initializing [ 3290.721623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3290.765784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3290.780181] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3294.991398] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3304.242616] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3309.705700] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3317.235884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3317.572724] Lustre: lustre-OST0000: new disk, initializing [ 3317.580419] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3317.586958] Lustre: 168038:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3317.680460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3324.193291] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3324.525831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3324.544807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3324.704637] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3334.412553] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3339.618660] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3339.819222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3344.718472] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3344.891395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3350.499645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3350.499980] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3350.501736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3350.521079] Lustre: Skipped 1 previous similar message [ 3350.546859] LustreError: Skipped 1 previous similar message [ 3352.783746] Lustre: server umount lustre-OST0000 complete [ 3357.155110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3357.165428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3357.182204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3357.188471] Lustre: Skipped 1 previous similar message [ 3362.673478] Lustre: server umount lustre-MDT0000 complete [ 3365.863974] LustreError: 166084:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3365.885798] LustreError: 166084:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3366.135201] LustreError: 166867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788272568 with bad export cookie 10976961200842380118 [ 3366.136390] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3366.145612] LustreError: 166867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3366.552748] Lustre: server umount lustre-MDT0001 complete [ 3374.145714] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:22:55 (1788272575) [ 3390.097312] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 3399.847478] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 3431.992607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3432.203691] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3432.250881] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3432.346982] Lustre: lustre-MDT0000: new disk, initializing [ 3432.433382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3432.451571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3436.575922] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3447.688753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3447.793696] Lustre: 172796:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3448.072814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3452.275504] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3461.527808] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3467.142932] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3474.406522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3474.736212] Lustre: lustre-OST0000: new disk, initializing [ 3474.739645] Lustre: Skipped 1 previous similar message [ 3474.743837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3474.748339] Lustre: Skipped 2 previous similar messages [ 3474.752746] Lustre: 174774:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3480.165996] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3482.166296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3482.177435] Lustre: Skipped 1 previous similar message [ 3482.185177] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3482.294246] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3488.671546] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3497.957207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3497.960113] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3497.970424] LustreError: Skipped 1 previous similar message [ 3497.971290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3497.971294] Lustre: Skipped 2 previous similar messages [ 3498.010358] Lustre: Skipped 2 previous similar messages [ 3509.216128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3509.434800] Lustre: server umount lustre-OST0000 complete [ 3514.860667] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3514.871900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3514.898522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3514.903281] Lustre: Skipped 5 previous similar messages [ 3519.644428] Lustre: server umount lustre-MDT0000 complete [ 3523.371742] LustreError: 172787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788272726 with bad export cookie 10976961200842381000 [ 3523.380589] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3523.386965] LustreError: 172787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3524.062564] Lustre: server umount lustre-MDT0001 complete [ 3532.574187] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:25:34 (1788272734) [ 3549.279242] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 3558.390762] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 3592.011605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3592.247729] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3592.286890] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3592.366519] Lustre: lustre-MDT0000: new disk, initializing [ 3592.455535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3592.461398] Lustre: Skipped 2 previous similar messages [ 3592.473474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3596.629315] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3607.035798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3607.231142] Lustre: 179097:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3607.604412] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3612.224589] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3620.205274] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3625.421690] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3631.773276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3632.087570] Lustre: 181073:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3637.497245] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3638.130279] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3647.667706] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3658.214178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3658.230909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3658.239650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3658.241945] Lustre: Skipped 2 previous similar messages [ 3658.251745] Lustre: Skipped 2 previous similar messages [ 3660.646582] Lustre: server umount lustre-OST0000 complete [ 3671.273024] Lustre: server umount lustre-MDT0000 complete [ 3673.569575] LustreError: 179109:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3673.593564] LustreError: 179109:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3674.955388] LustreError: 179090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788272877 with bad export cookie 10976961200842382085 [ 3674.958223] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3674.971253] LustreError: 179090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3675.279309] Lustre: server umount lustre-MDT0001 complete [ 3685.382365] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 3688.466225] Key type lgssc unregistered [ 3688.729441] LNet: 182933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3688.737982] LNetError: 182933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3688.764069] LNet: Removed LNI 192.168.203.130@tcp [ 3689.675096] Key type .llcrypt unregistered [ 3689.681121] Key type ._llcrypt unregistered [ 3713.005974] Key type ._llcrypt registered [ 3713.008938] Key type .llcrypt registered [ 3713.143426] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 3714.413946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3714.636745] alg: No test for adler32 (adler32-zlib) [ 3715.771501] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 3716.035601] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 3717.728299] Key type lgssc registered [ 3718.881875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3726.113807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3726.139927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3727.567859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3731.221540] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3738.086050] LustreError: 184350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3738.106491] LustreError: 184350:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3738.489052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3738.811489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3742.662545] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3749.164892] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3753.822566] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3760.151786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3760.536527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3766.702519] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3772.885026] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3774.124447] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3795.435210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3795.441675] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3795.451482] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3798.517446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3798.526701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3798.527170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3798.557524] Lustre: Skipped 1 previous similar message [ 3799.826638] Lustre: server umount lustre-OST0000 complete [ 3803.619907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3803.621874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3803.628453] Lustre: Skipped 1 previous similar message [ 3803.634429] Lustre: Skipped 1 previous similar message [ 3808.739978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3808.746216] Lustre: Skipped 1 previous similar message [ 3809.861398] Lustre: server umount lustre-MDT0000 complete [ 3813.860956] LustreError: 184347:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3813.888925] LustreError: 184347:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3814.670402] LustreError: 184329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788273017 with bad export cookie 11263497343262443990 [ 3814.685884] LustreError: 184329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3814.686525] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3815.301525] Lustre: server umount lustre-MDT0001 complete [ 3829.818638] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 3833.431840] Key type lgssc unregistered [ 3833.773807] LNet: 187904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3833.787583] LNetError: 187904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3833.813696] LNet: Removed LNI 192.168.203.130@tcp [ 3834.910484] Key type .llcrypt unregistered [ 3834.925954] Key type ._llcrypt unregistered [ 3861.178734] Key type ._llcrypt registered [ 3861.180632] Key type .llcrypt registered [ 3861.288432] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 3862.370812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3862.573576] alg: No test for adler32 (adler32-zlib) [ 3863.674983] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 3863.987992] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 3865.672165] Key type lgssc registered [ 3866.964560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3875.335973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3875.369595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3876.830695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3881.649606] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3887.075745] LustreError: 189325:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3887.111363] LustreError: 189325:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3890.700240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3891.064751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3895.926803] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3904.047837] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3909.471059] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3915.762868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3916.184404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3921.660629] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3922.355326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3929.455560] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3940.836279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3940.839625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_create to node 0@lo failed: rc = -107 [ 3940.864126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3940.883193] LustreError: Skipped 1 previous similar message [ 3942.639089] Lustre: lustre-OST0000: Not available for connect from 192.168.203.30@tcp (stopping) [ 3942.658829] Lustre: Skipped 1 previous similar message [ 3945.959882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3945.975713] Lustre: Skipped 1 previous similar message [ 3946.581966] Lustre: server umount lustre-OST0000 complete [ 3955.578449] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8ca291416000.ost_server_uuid 50 [ 3957.682513] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3973.090399] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3973.098671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3973.118407] Lustre: Skipped 1 previous similar message [ 3973.131326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3975.651287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3977.856859] Lustre: server umount lustre-MDT0000 complete [ 3980.769502] LustreError: 189319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3980.793479] LustreError: 189319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3981.571425] LustreError: 189304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788273184 with bad export cookie 48924755817592280 [ 3981.592781] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3981.598682] LustreError: 190053:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 3981.625934] LustreError: 190053:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 3982.022144] Lustre: server umount lustre-MDT0001 complete [ 3990.882745] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:33:12 (1788273192) [ 3997.622979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3998.052236] LustreError: 193239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3998.192743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4002.617978] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4003.300782] LustreError: 193240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4007.395654] LustreError: 193239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4011.212426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4011.652437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4015.969607] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4022.823461] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4027.482193] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4033.433540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4033.731170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4039.161251] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4043.001376] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4045.919313] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4052.424821] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4052.679771] Lustre: lustre-OST0001: new disk, initializing [ 4052.685562] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4052.692950] Lustre: 196081:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4052.778791] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4058.400246] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4060.752284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4060.764920] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4060.847544] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4067.843171] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4071.398195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4071.422695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4071.430870] Lustre: Skipped 3 previous similar messages [ 4076.661708] Lustre: server umount lustre-OST0000 complete [ 4078.820824] LustreError: 195063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4082.256532] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4082.487631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4086.372726] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4086.578856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4097.000831] LustreError: 195063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4097.019995] LustreError: 195063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 4102.113665] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4102.126196] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4102.134669] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4102.142826] Lustre: Skipped 2 previous similar messages [ 4102.151295] Lustre: Skipped 4 previous similar messages [ 4105.891069] Lustre: server umount lustre-OST0001 complete [ 4112.358791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4112.376442] Lustre: Skipped 1 previous similar message [ 4115.883882] Lustre: server umount lustre-MDT0000 complete [ 4119.248246] LustreError: 193221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788273321 with bad export cookie 48924755817593351 [ 4119.249485] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4119.256939] LustreError: 193221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4119.579337] Lustre: server umount lustre-MDT0001 complete [ 4141.186592] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:35:43 (1788273343) [ 4147.577579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4147.807865] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4147.850937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4148.044624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4152.167401] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4159.703219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4159.818172] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4159.860201] Lustre: 199519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4164.302335] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4171.045399] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4176.179657] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4181.986763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4182.176644] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4188.382326] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4191.389394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 4195.287315] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4201.701724] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4201.808700] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4207.433077] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4208.254736] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 4213.605524] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4217.314967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4217.316683] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4217.334481] LustreError: Skipped 2 previous similar messages [ 4217.335333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4217.335337] Lustre: Skipped 2 previous similar messages [ 4217.377371] Lustre: Skipped 1 previous similar message [ 4221.981963] Lustre: server umount lustre-OST0000 complete [ 4223.974301] LustreError: 201372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4224.003102] LustreError: 201372:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4238.818160] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4243.187627] Lustre: server umount lustre-OST0001 complete [ 4247.009668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4249.578539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4249.580564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4249.594454] Lustre: Skipped 3 previous similar messages [ 4249.609116] Lustre: Skipped 9 previous similar messages [ 4252.885816] Lustre: server umount lustre-MDT0000 complete [ 4256.468854] LustreError: 199511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788273459 with bad export cookie 48924755817594758 [ 4256.482815] LustreError: 199511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4256.484270] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4256.826042] Lustre: server umount lustre-MDT0001 complete [ 4278.027208] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:37:59 (1788273479) [ 4292.948358] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 4302.272758] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 4333.341243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4333.516141] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4333.540689] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4333.607376] Lustre: lustre-MDT0000: new disk, initializing [ 4333.669766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4333.674422] Lustre: Skipped 3 previous similar messages [ 4333.683247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4337.468127] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4346.413885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4346.512620] Lustre: 207622:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4346.559378] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4346.563254] Lustre: Skipped 1 previous similar message [ 4346.628187] Lustre: lustre-MDT0001: new disk, initializing [ 4346.779441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4346.785785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4350.614183] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4358.630946] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4363.174851] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4368.826847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4369.059556] Lustre: lustre-OST0000: new disk, initializing [ 4369.063459] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4369.068484] Lustre: 209588:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4375.328645] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4375.655047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4375.666560] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4375.891061] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4385.487515] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4390.537974] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4390.877730] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4396.575693] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4396.837240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4401.635285] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4401.636761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4401.643793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4401.645906] Lustre: Skipped 1 previous similar message [ 4401.654054] Lustre: Skipped 1 previous similar message [ 4405.094688] Lustre: server umount lustre-OST0000 complete [ 4416.770103] Lustre: server umount lustre-MDT0000 complete [ 4416.993607] LustreError: 207635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4417.004172] LustreError: 207635:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 4420.190804] LustreError: 207616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788273622 with bad export cookie 48924755817596543 [ 4420.198165] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4420.207520] LustreError: 207616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4420.558992] Lustre: server umount lustre-MDT0001 complete [ 4427.338461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4432.223022] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4439.217267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4443.333446] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4449.359217] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4454.108478] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4459.681345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4465.250096] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4470.936721] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4474.605942] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4474.770497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4478.640825] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4478.847080] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4480.481409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4480.487485] LustreError: Skipped 1 previous similar message [ 4486.693799] Lustre: server umount lustre-OST0000 complete [ 4491.885924] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4496.338384] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4524.070767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4524.321158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4524.327524] Lustre: Skipped 5 previous similar messages [ 4530.008839] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4533.547743] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4533.567348] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4536.082797] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4542.733923] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 4547.707762] Lustre: server umount lustre-OST0000 complete [ 4552.161664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4552.168974] LustreError: Skipped 1 previous similar message [ 4552.172721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4552.180563] Lustre: Skipped 4 previous similar messages [ 4552.184555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4552.187978] Lustre: Skipped 5 previous similar messages [ 4556.733419] Lustre: server umount lustre-MDT0000 complete [ 4558.818586] LustreError: 212534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4558.835356] LustreError: 212534:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4559.386632] LustreError: 211787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788273762 with bad export cookie 48924755817597425 [ 4559.393278] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4559.404222] LustreError: 211787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4569.677933] Lustre: server umount lustre-MDT0001 complete [ 4576.397453] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:42:58 (1788273778) [ 4582.069657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4585.986525] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4592.751771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4596.365185] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4601.433250] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4605.202253] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4609.764361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4613.579586] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4618.551487] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4621.925587] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4629.493335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 4633.299865] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4633.494640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4639.202640] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4639.365747] Lustre: lustre-OST0001: new disk, initializing [ 4639.369100] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4639.374369] Lustre: 221184:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4643.555915] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4649.538415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4649.546818] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4649.591050] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4650.199226] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4652.999254] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4653.121580] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4656.137453] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4656.298705] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4663.968925] Lustre: server umount lustre-OST0001 complete [ 4668.530064] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4668.688744] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4671.987687] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4672.212318] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4698.133784] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4702.406807] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4703.729765] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4703.736836] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 4703.737115] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4708.148707] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4709.539668] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4713.969636] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4713.979311] LustreError: Skipped 1 previous similar message [ 4717.512342] Lustre: server umount lustre-OST0001 complete [ 4738.226047] Lustre: server umount lustre-OST0000 complete [ 4746.596521] Lustre: server umount lustre-MDT0000 complete [ 4748.990411] LustreError: 217749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788273951 with bad export cookie 48924755817598384 [ 4748.998241] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4749.001167] LustreError: 217749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4755.273116] Lustre: server umount lustre-MDT0001 complete [ 4771.712803] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:46:13 (1788273973) [ 4776.087451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4776.297421] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4776.315183] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4779.464259] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4784.521942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4784.617417] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4784.642472] Lustre: 226422:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4784.817837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4784.823984] Lustre: Skipped 6 previous similar messages [ 4787.477953] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4792.397751] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4796.086715] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4800.366352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4800.479944] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4804.170626] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4808.571504] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4814.727703] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4814.795811] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4819.049077] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4823.564664] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4824.469401] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4826.973909] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4827.128867] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4829.797309] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4829.934295] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4833.363828] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ca2a07a2000.ost_server_uuid 50 [ 4834.306138] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8ca2a07a2000.ost_server_uuid in FULL state after 0 sec [ 4835.104289] Lustre: Permanently deactivating lustre-OST0001 [ 4835.118990] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 4837.533930] Lustre: Permanently reactivating lustre-OST0001 [ 4842.392351] Lustre: server umount lustre-OST0001 complete [ 4842.469383] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4842.477378] LustreError: 228266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4842.478779] Lustre: Skipped 10 previous similar messages [ 4842.487044] LustreError: 228266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 30 previous similar messages [ 4846.727196] Lustre: server umount lustre-OST0000 complete [ 4851.685334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4851.688255] Lustre: Skipped 12 previous similar messages [ 4858.905209] Lustre: server umount lustre-MDT0000 complete [ 4861.119810] LustreError: 226458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788274063 with bad export cookie 48924755817599672 [ 4861.123539] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4861.128260] LustreError: 226458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4870.734482] Lustre: server umount lustre-MDT0001 complete [ 4886.321917] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:48:08 (1788274088) [ 4890.845810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4890.997701] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4891.016247] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4891.023706] Lustre: Skipped 5 previous similar messages [ 4893.632769] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4898.918835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4898.991888] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4899.010376] Lustre: 233108:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4902.133683] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4908.303359] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4917.867058] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4929.648382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4930.084398] Lustre: Permanently deactivating lustre-OST0000 [ 4930.133908] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 4936.318553] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 4940.153502] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4950.880985] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4958.646777] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4958.706742] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4958.712084] Lustre: Skipped 1 previous similar message [ 4964.739300] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4968.631202] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 4971.313668] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4973.263604] Lustre: Permanently reactivating lustre-OST0000 [ 4973.283089] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 4973.291052] Lustre: Skipped 2 previous similar messages [ 4976.245043] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 4976.246408] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 39 seconds. I think it's dead, and I am evicting it. exp 000000006b641dc2, cur 1788274178 deadline 1788274169 last 1788274139 [ 4976.251483] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4976.261811] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4976.295483] Lustre: Skipped 1 previous similar message [ 4976.382806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4989.415238] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4989.426716] LustreError: Skipped 4 previous similar messages [ 4989.796620] Lustre: server umount lustre-OST0001 complete [ 5007.328139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5007.456986] Lustre: server umount lustre-OST0000 complete [ 5016.898718] Lustre: server umount lustre-MDT0000 complete [ 5020.427691] LustreError: 233101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788274223 with bad export cookie 48924755817601317 [ 5020.434746] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5020.439313] LustreError: 233101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5020.822699] Lustre: server umount lustre-MDT0001 complete [ 5029.657864] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 5032.019086] Key type lgssc unregistered [ 5032.308695] LNet: 237992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5032.314935] LNetError: 237992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5032.328442] LNet: Removed LNI 192.168.203.130@tcp [ 5033.096996] Key type .llcrypt unregistered [ 5033.100087] Key type ._llcrypt unregistered [ 5051.286494] Key type ._llcrypt registered [ 5051.288513] Key type .llcrypt registered [ 5051.363817] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 5052.136987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5052.161205] alg: No test for adler32 (adler32-zlib) [ 5053.219094] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 5053.445433] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 5055.104466] Key type lgssc registered [ 5056.139979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5064.805201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5064.824483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5066.149469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5069.830483] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5075.427898] LustreError: 239438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5075.443083] LustreError: 239438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5077.235948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5077.339574] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5077.364297] Lustre: 239427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5077.376219] Lustre: Permanently deactivating lustre-MDT0001 [ 5077.387652] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5077.683421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5080.542726] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5080.553397] LustreError: 239469:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5080.573792] LustreError: 239469:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5080.597364] LustreError: 239469:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 5081.783633] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5088.504708] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5092.585388] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5097.818037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5098.115404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5103.274815] LustreError: 241278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5103.303273] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 5103.481430] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5108.705092] LustreError: 241278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5109.404456] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5110.760216] LustreError: 241621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5110.782587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5110.790059] LustreError: 241621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5114.637017] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5114.885734] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5119.992718] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5119.996081] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5120.311338] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5125.716098] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5129.162809] Lustre: Permanently reactivating lustre-MDT0001 [ 5129.181708] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5129.190851] Lustre: Skipped 2 previous similar messages [ 5132.540172] Lustre: lustre-MDT0001: Client 43619357-7b01-4a9a-aeab-c2c04d772df8 (at 192.168.203.30@tcp) reconnecting [ 5138.703184] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5140.232890] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5145.147971] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5145.393042] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5149.921360] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5150.105847] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5152.219701] Lustre: Permanently deactivating lustre-MDT0001 [ 5152.232167] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5152.241201] Lustre: Skipped 2 previous similar messages [ 5161.095180] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5171.178633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5171.200175] Lustre: Skipped 1 previous similar message [ 5171.203965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5171.221967] Lustre: Skipped 3 previous similar messages [ 5176.304536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5176.312982] Lustre: Skipped 3 previous similar messages [ 5177.251729] Lustre: server umount lustre-MDT0000 complete [ 5181.232609] LustreError: 239420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788274383 with bad export cookie 14751502051114120092 [ 5181.240212] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5181.253719] LustreError: 239420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5181.424089] LustreError: 239434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5181.443487] LustreError: 239434:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5181.553840] Lustre: server umount lustre-MDT0001 complete [ 5187.318452] Lustre: server umount lustre-OST0000 complete [ 5196.936242] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:53:18 (1788274398) [ 5207.017670] Lustre: server umount lustre-OST0001 complete [ 5214.344187] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 5223.982813] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 5261.961077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5262.144812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5262.151477] Lustre: Skipped 2 previous similar messages [ 5262.169647] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5262.235921] Lustre: lustre-MDT0000: new disk, initializing [ 5262.293302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5262.306995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5265.692721] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5276.311925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5276.555545] Lustre: 247744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5276.632436] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5276.636891] Lustre: Skipped 1 previous similar message [ 5276.803995] Lustre: lustre-MDT0001: new disk, initializing [ 5276.969491] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5277.066198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5277.098339] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5282.151971] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5294.111596] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5299.857362] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5305.197581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5305.504582] Lustre: lustre-OST0000: new disk, initializing [ 5305.507652] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5305.512464] Lustre: 249714:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5305.590681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5311.445860] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5311.570462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5311.579123] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5311.631618] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5320.132218] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5322.262202] LustreError: 247752:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5330.226051] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5330.433610] Lustre: lustre-OST0001: new disk, initializing [ 5330.436957] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5330.441230] Lustre: 250826:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5332.283798] LustreError: 247752:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5332.319478] LustreError: 249511:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5336.580487] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5339.345375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5339.354312] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5339.445517] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5342.379565] LustreError: 249511:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5346.306930] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5347.948530] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5352.725432] Lustre: lustre-OST0001: Not available for connect from 192.168.203.30@tcp (stopping) [ 5354.980188] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5354.980374] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5354.987486] LustreError: Skipped 1 previous similar message [ 5354.994328] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5354.996472] Lustre: Skipped 2 previous similar messages [ 5355.022044] Lustre: Skipped 1 previous similar message [ 5356.756610] Lustre: server umount lustre-OST0001 complete [ 5357.802142] LustreError: 251499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5375.466556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5375.470784] LustreError: 249699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5375.476622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5375.486933] Lustre: Skipped 1 previous similar message [ 5375.513166] LustreError: 249699:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 5385.700481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5385.707521] Lustre: Skipped 4 previous similar messages [ 5388.257589] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5388.450646] Lustre: server umount lustre-OST0000 complete [ 5394.917585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5394.935991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5399.312574] Lustre: server umount lustre-MDT0000 complete [ 5403.535564] LustreError: 248538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788274606 with bad export cookie 14751502051114123123 [ 5403.557260] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5409.995962] Lustre: server umount lustre-MDT0001 complete [ 5426.068455] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 5429.791242] Key type lgssc unregistered [ 5430.128805] LNet: 252865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5430.137709] LNetError: 252865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5430.163074] LNet: Removed LNI 192.168.203.130@tcp [ 5430.932455] Key type .llcrypt unregistered [ 5430.935247] Key type ._llcrypt unregistered [ 5462.223585] Key type ._llcrypt registered [ 5462.225658] Key type .llcrypt registered [ 5462.309681] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 5463.385564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5463.414271] alg: No test for adler32 (adler32-zlib) [ 5464.429550] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 5464.732722] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 5466.408160] Key type lgssc registered [ 5467.359863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5474.552129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5474.570159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5475.857634] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5475.877745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5475.999737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5479.798097] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5486.809254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5486.909955] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5486.946022] Lustre: 255054:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5487.189885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5491.353968] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5497.801445] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5503.635810] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5510.937314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5511.119932] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5511.352203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5519.413718] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5519.589191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 5527.504518] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5581.283241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5581.291103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5581.325499] Lustre: Skipped 1 previous similar message [ 5582.773391] Lustre: server umount lustre-OST0000 complete [ 5588.819080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5606.331319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5606.789595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5613.260975] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5620.296927] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5622.127732] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.140664] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5622.150055] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5622.167350] Lustre: Skipped 1 previous similar message [ 5663.202064] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5663.207365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5663.217909] LustreError: Skipped 1 previous similar message [ 5663.236354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5663.243841] Lustre: Skipped 1 previous similar message [ 5666.887732] Lustre: server umount lustre-OST0000 complete [ 5671.395534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5671.410694] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5671.418227] Lustre: Skipped 1 previous similar message [ 5671.422606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5671.425921] Lustre: Skipped 1 previous similar message [ 5673.444816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5673.446263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5673.477305] Lustre: Skipped 1 previous similar message [ 5676.907767] Lustre: server umount lustre-MDT0000 complete [ 5680.304097] LustreError: 255045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788274882 with bad export cookie 7132276683581871664 [ 5680.314574] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5680.319191] LustreError: 255045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5680.625198] Lustre: server umount lustre-MDT0001 complete [ 5691.601985] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 5694.236539] Key type lgssc unregistered [ 5694.513823] LNet: 260493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5694.522751] LNetError: 260493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5694.545968] LNet: Removed LNI 192.168.203.130@tcp [ 5695.192237] Key type .llcrypt unregistered [ 5695.194955] Key type ._llcrypt unregistered [ 5720.839905] Key type ._llcrypt registered [ 5720.841913] Key type .llcrypt registered [ 5720.926752] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 5721.624241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5721.642634] alg: No test for adler32 (adler32-zlib) [ 5722.759657] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 5723.029154] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 5724.691511] Key type lgssc registered [ 5725.660404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5732.621228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5732.639308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5734.056699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5737.545350] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5743.595213] LustreError: 261911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5743.607097] LustreError: 261911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5745.771110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5746.116263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5750.851335] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5759.812649] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5766.870604] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5776.141945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5776.687888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5778.743272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 5784.081749] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5792.295026] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5814.760510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5814.774603] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5814.791654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5816.805056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5816.810388] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5816.813581] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5821.022475] Lustre: server umount lustre-OST0000 complete [ 5827.047483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5827.051084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5827.069989] Lustre: Skipped 1 previous similar message [ 5827.075339] Lustre: Skipped 1 previous similar message [ 5832.163040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5832.176514] Lustre: Skipped 2 previous similar messages [ 5837.291609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5837.296720] Lustre: Skipped 1 previous similar message [ 5840.355035] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5840.456771] Lustre: server umount lustre-MDT0000 complete [ 5842.414807] LustreError: 261911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5842.443256] LustreError: 261911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5843.976952] LustreError: 261891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788275046 with bad export cookie 3964751583194846930 [ 5843.991151] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5844.262366] Lustre: server umount lustre-MDT0001 complete [ 5853.561627] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 5856.224989] Key type lgssc unregistered [ 5856.509444] LNet: 266223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5856.515713] LNetError: 266223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5856.540182] LNet: Removed LNI 192.168.203.130@tcp [ 5857.376391] Key type .llcrypt unregistered [ 5857.380801] Key type ._llcrypt unregistered [ 5875.837612] Key type ._llcrypt registered [ 5875.841860] Key type .llcrypt registered [ 5875.926355] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 5876.683539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5876.710932] alg: No test for adler32 (adler32-zlib) [ 5877.908194] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 5878.194376] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 5879.867137] Key type lgssc registered [ 5881.840322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5890.549420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5890.595662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5892.197983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5896.937391] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5901.797698] LustreError: 267453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5901.816625] LustreError: 267453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5906.914887] LustreError: 267454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5906.993252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5907.424622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5913.113929] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5919.487733] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5924.114127] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5930.363031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5930.756733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5936.018899] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5936.889144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5942.793751] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5957.601293] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5957.605603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5957.616944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5957.626043] Lustre: Skipped 1 previous similar message [ 5957.642403] Lustre: Skipped 1 previous similar message [ 5960.139962] Lustre: server umount lustre-OST0000 complete [ 5967.845027] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5967.879722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5967.886907] Lustre: Skipped 1 previous similar message [ 5971.372645] Lustre: server umount lustre-MDT0000 complete [ 5972.967496] LustreError: 268183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5972.991896] LustreError: 268183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5975.973920] LustreError: 267432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788275178 with bad export cookie 3333743321761998504 [ 5975.983362] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5975.999285] LustreError: 267432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5976.299686] Lustre: server umount lustre-MDT0001 complete [ 5986.879785] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 5989.384441] Key type lgssc unregistered [ 5989.661869] LNet: 271133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5989.678864] LNetError: 271133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5990.761578] LNet: Removed LNI 192.168.203.130@tcp [ 5991.653689] Key type .llcrypt unregistered [ 5991.655984] Key type ._llcrypt unregistered [ 6012.795987] Key type ._llcrypt registered [ 6012.801711] Key type .llcrypt registered [ 6012.942658] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 6013.792563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6013.881419] alg: No test for adler32 (adler32-zlib) [ 6014.857363] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 6015.029061] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 6016.680167] Key type lgssc registered [ 6017.676434] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6024.496521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6024.529781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6025.860722] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6029.959302] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6035.432661] LustreError: 272549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6035.455582] LustreError: 272549:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6038.120976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6038.517204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6043.041587] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6051.521502] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6057.824706] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6067.020248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6067.580966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6070.660259] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6075.447960] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6084.663346] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6109.665318] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6109.670677] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6109.674046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6109.702085] Lustre: Skipped 1 previous similar message [ 6114.787181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6114.802914] Lustre: Skipped 2 previous similar messages [ 6115.306310] Lustre: server umount lustre-OST0000 complete [ 6119.911074] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6119.925420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6119.929095] Lustre: Skipped 1 previous similar message [ 6124.799144] Lustre: server umount lustre-MDT0000 complete [ 6125.032342] LustreError: 274392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6125.051770] LustreError: 274392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6127.654946] LustreError: 272529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788275330 with bad export cookie 4068279448344681439 [ 6127.661604] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6127.667141] LustreError: 272529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6127.879719] Lustre: server umount lustre-MDT0001 complete [ 6135.882602] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 6138.873090] Key type lgssc unregistered [ 6139.172598] LNet: 276862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6139.187933] LNetError: 276862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6140.263710] LNet: Removed LNI 192.168.203.130@tcp [ 6141.147338] Key type .llcrypt unregistered [ 6141.153251] Key type ._llcrypt unregistered [ 6158.752977] Key type ._llcrypt registered [ 6158.755560] Key type .llcrypt registered [ 6158.856614] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 6160.078332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6160.096301] alg: No test for adler32 (adler32-zlib) [ 6161.189577] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 6161.502991] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 6163.146204] Key type lgssc registered [ 6164.414847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6172.240465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6172.268887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6173.766914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6178.055035] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6183.910896] LustreError: 278097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6183.942433] LustreError: 278097:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6185.737708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6186.147648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6190.376845] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6197.392700] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6203.229124] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6211.958675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6212.544739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6218.751071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6220.534565] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6227.736332] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6244.321843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6244.328990] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6244.345337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6246.380814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6246.380974] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6246.381032] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6246.411671] Lustre: Skipped 1 previous similar message [ 6248.946975] Lustre: server umount lustre-OST0000 complete [ 6256.610424] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6256.614603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6256.619079] Lustre: Skipped 1 previous similar message [ 6256.627695] Lustre: Skipped 1 previous similar message [ 6259.943292] Lustre: server umount lustre-MDT0000 complete [ 6261.731220] LustreError: 278106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6261.742551] LustreError: 278106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6264.094240] LustreError: 278064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788275466 with bad export cookie 11096357242228189576 [ 6264.120671] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6264.593795] Lustre: server umount lustre-MDT0001 complete [ 6278.097968] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 6280.530564] Key type lgssc unregistered [ 6280.717816] LNet: 281802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6280.723579] LNetError: 281802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6280.735562] LNet: Removed LNI 192.168.203.130@tcp [ 6281.475780] Key type .llcrypt unregistered [ 6281.478706] Key type ._llcrypt unregistered [ 6292.652905] Key type ._llcrypt registered [ 6292.656937] Key type .llcrypt registered [ 6292.805881] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6313.404109] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 6331.078520] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 6332.602433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6332.612613] alg: No test for adler32 (adler32-zlib) [ 6333.600300] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 6333.800737] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 6335.457672] Key type lgssc registered [ 6336.621651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6367.168504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6367.218665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6368.516283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6368.562663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6368.655889] Lustre: lustre-MDT0000: new disk, initializing [ 6368.704512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6368.715255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6373.026150] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6382.592927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6382.698486] Lustre: 285356:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6382.743086] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6382.747927] Lustre: Skipped 1 previous similar message [ 6382.827349] Lustre: lustre-MDT0001: new disk, initializing [ 6382.940451] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6383.002295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6383.013964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6388.003733] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6397.152843] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6402.671103] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6409.644115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6409.912924] Lustre: lustre-OST0000: new disk, initializing [ 6409.916674] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6409.923411] Lustre: 287328:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6410.026681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6417.967266] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6419.094037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6419.119171] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6419.276017] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6429.441162] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6435.867242] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6436.220426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6441.578877] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6441.940625] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6445.033512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6445.043091] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6445.056101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6445.073422] Lustre: Skipped 1 previous similar message [ 6450.150556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6450.161894] Lustre: Skipped 1 previous similar message [ 6450.332208] Lustre: server umount lustre-OST0000 complete [ 6454.755717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6454.767035] LustreError: Skipped 1 previous similar message [ 6454.775300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6454.797798] Lustre: Skipped 1 previous similar message [ 6454.805425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6460.386133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6460.399334] Lustre: Skipped 3 previous similar messages [ 6460.731699] Lustre: server umount lustre-MDT0000 complete [ 6464.687758] LustreError: 285350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788275667 with bad export cookie 16263095632657806509 [ 6464.699662] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6464.705603] LustreError: 285350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6465.164802] Lustre: server umount lustre-MDT0001 complete [ 6476.934614] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:14:37 (1788275677) [ 6485.201600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6485.896416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6490.208626] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6491.106133] LustreError: 289740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6491.129934] LustreError: 289740:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6496.227289] LustreError: 289741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6498.530403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6498.834366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6504.432357] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6513.101380] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6519.329148] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6526.358366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6526.722878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6532.949550] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6541.420820] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6556.642517] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6556.648167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6556.655332] Lustre: Skipped 1 previous similar message [ 6556.658913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6559.722879] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6559.737070] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6560.913411] Lustre: server umount lustre-OST0000 complete [ 6564.842733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6564.886939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6564.896898] Lustre: Skipped 2 previous similar messages [ 6571.447293] Lustre: server umount lustre-MDT0000 complete [ 6575.080744] LustreError: 289741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6575.111938] LustreError: 289741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6576.070770] LustreError: 290475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788275778 with bad export cookie 16263095632657807398 [ 6576.082171] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6576.085254] LustreError: 290475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6577.002652] Lustre: server umount lustre-MDT0001 complete [ 6586.993480] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 6589.918855] Key type lgssc unregistered [ 6590.239621] LNet: 293299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6590.262527] LNetError: 293299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6590.303318] LNet: Removed LNI 192.168.203.130@tcp [ 6591.274510] Key type .llcrypt unregistered [ 6591.279203] Key type ._llcrypt unregistered [ 6619.765469] Key type ._llcrypt registered [ 6619.768279] Key type .llcrypt registered [ 6619.901852] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 6633.725408] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 6634.434329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6634.481606] alg: No test for adler32 (adler32-zlib) [ 6635.530385] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 6635.742646] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 6637.368135] Key type lgssc registered [ 6638.264873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6689.595113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6689.609433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6690.759943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6690.789153] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6690.841367] Lustre: lustre-MDT0000: new disk, initializing [ 6690.913506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6690.930754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6697.132049] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6708.068757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6708.204422] Lustre: 297127:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6708.273120] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6708.277909] Lustre: Skipped 1 previous similar message [ 6708.358464] Lustre: lustre-MDT0001: new disk, initializing [ 6708.493830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6708.556713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6708.576502] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6713.227759] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6723.601686] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6731.726057] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6739.795536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6740.239644] Lustre: lustre-OST2710: new disk, initializing [ 6740.243840] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 6740.258632] Lustre: 299100:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6740.362860] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6745.701856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6745.717685] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6745.850741] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 6747.367103] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6756.750979] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6762.905621] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6763.093916] Lustre: lustre-OST03e8: new disk, initializing [ 6763.096753] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 6763.100440] Lustre: 300163:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6763.162501] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6769.336633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6769.349719] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6769.413323] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 6769.709510] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6779.214961] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6788.007245] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6788.468165] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6793.761590] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 6794.006643] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6799.743892] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6800.081612] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6806.731708] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 6806.940665] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6813.614970] Lustre: server umount lustre-MDT0000 complete [ 6813.668423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6813.675032] Lustre: Skipped 1 previous similar message [ 6817.439430] LustreError: 297122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788276020 with bad export cookie 14999620502210965752 [ 6817.441113] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6817.444128] LustreError: 297122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6817.828307] Lustre: server umount lustre-MDT0001 complete [ 6832.415526] Lustre: server umount lustre-OST2710 complete [ 6834.976571] Lustre: 294925:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788276021/real 1788276021] req@0000000026ac551b x1875143125179520/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788276037 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6835.006528] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6835.015714] Lustre: Skipped 2 previous similar messages [ 6836.205536] Lustre: server umount lustre-OST03e8 complete [ 6855.628845] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 6866.045800] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 6900.267895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6900.529481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6900.563883] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6900.635710] Lustre: lustre-MDT0000: new disk, initializing [ 6900.735256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6900.757471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6905.667765] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6916.021847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6916.141423] Lustre: 305412:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6916.194854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6916.198509] Lustre: Skipped 1 previous similar message [ 6916.253577] Lustre: lustre-MDT0001: new disk, initializing [ 6916.318433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6916.355383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6916.372573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6921.538196] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6932.613329] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6939.750427] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6946.758273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6947.087458] Lustre: lustre-OST0000: new disk, initializing [ 6947.091923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6947.103359] Lustre: 307387:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6947.228535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6954.075697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6954.084330] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6954.169422] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6954.266070] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6963.811346] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6968.980147] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6969.212851] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6973.861738] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6974.127447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6980.074935] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6980.075208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6980.083043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6980.095049] Lustre: Skipped 1 previous similar message [ 6982.530391] Lustre: server umount lustre-OST0000 complete [ 6988.259276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6988.264093] LustreError: Skipped 1 previous similar message [ 6988.271535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6988.283090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6988.290076] Lustre: Skipped 1 previous similar message [ 6990.308102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6990.317646] Lustre: Skipped 1 previous similar message [ 6992.813332] Lustre: server umount lustre-MDT0000 complete [ 6995.425873] LustreError: 306519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6995.451939] LustreError: 306519:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 6996.466142] LustreError: 305406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788276199 with bad export cookie 14999620502210967439 [ 6996.467771] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6996.476486] LustreError: 305406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6996.858641] Lustre: server umount lustre-MDT0001 complete [ 7004.904327] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:23:26 (1788276206) [ 7006.207269] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 7007.911859] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:23:29 (1788276209) [ 7011.473962] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 7038.764795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7039.051784] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7039.069298] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7039.277532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7043.781540] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7053.961245] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7054.068931] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7054.096397] Lustre: 311241:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7059.902991] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7081.343925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7081.620416] Lustre: Denying initial registration attempt from nid 192.168.203.130@tcp, specified as failover [ 7081.647675] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 7081.658555] LustreError: 312695:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 7081.674531] LustreError: 312695:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7081.684788] LustreError: 312695:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7081.729136] Lustre: server umount lustre-OST0000 complete [ 7081.732987] LustreError: 312695:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 7088.609992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7088.611714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7088.633924] Lustre: Skipped 2 previous similar messages [ 7088.647698] Lustre: Skipped 1 previous similar message [ 7093.731582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7093.739192] Lustre: Skipped 1 previous similar message [ 7094.485052] Lustre: server umount lustre-MDT0000 complete [ 7097.741696] LustreError: 311234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788276300 with bad export cookie 14999620502210968314 [ 7097.744360] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7097.751390] LustreError: 311234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7098.277425] Lustre: server umount lustre-MDT0001 complete [ 7107.893576] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 7110.750774] Key type lgssc unregistered [ 7111.000757] LNet: 313821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7111.015972] LNetError: 313821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7111.040941] LNet: Removed LNI 192.168.203.130@tcp [ 7111.884834] Key type .llcrypt unregistered [ 7111.886938] Key type ._llcrypt unregistered [ 7123.950528] Key type ._llcrypt registered [ 7123.956052] Key type .llcrypt registered [ 7124.095102] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 7125.163698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7125.179417] alg: No test for adler32 (adler32-zlib) [ 7126.342836] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 7126.647908] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 7128.344242] Key type lgssc registered [ 7129.343607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7173.943175] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 7184.877663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7184.899682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7186.209853] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7186.253616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7186.401134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7191.638452] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7201.505126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7201.806381] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7201.883828] Lustre: 316585:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7202.130457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7207.423491] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7215.206358] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7221.109849] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7240.106539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7240.327621] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7240.476669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7246.322485] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7253.106627] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7256.545829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7256.557412] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7256.567435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7257.063516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7257.071081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7257.081806] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7261.956929] Lustre: server umount lustre-OST0000 complete [ 7267.305461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7267.306983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7267.333295] Lustre: Skipped 1 previous similar message [ 7267.334930] Lustre: Skipped 1 previous similar message [ 7272.377683] Lustre: server umount lustre-MDT0000 complete [ 7275.997310] LustreError: 317337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788276478 with bad export cookie 11302368853186322318 [ 7276.003893] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7276.017680] LustreError: 317337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7276.340601] Lustre: server umount lustre-MDT0001 complete [ 7286.553374] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 7289.443859] Key type lgssc unregistered [ 7289.728839] LNet: 320162:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7289.740041] LNetError: 320162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7289.752220] LNet: Removed LNI 192.168.203.130@tcp [ 7290.623279] Key type .llcrypt unregistered [ 7290.625934] Key type ._llcrypt unregistered [ 7316.125546] Key type ._llcrypt registered [ 7316.128732] Key type .llcrypt registered [ 7316.286647] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 7317.680498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7317.818404] alg: No test for adler32 (adler32-zlib) [ 7319.106413] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 7319.367452] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 7321.064325] Key type lgssc registered [ 7322.374560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7329.542906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7329.571827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7331.118090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7336.004631] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7340.520517] LustreError: 321581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7340.544883] LustreError: 321581:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7344.636101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7345.076994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7349.445415] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7356.897429] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7362.267783] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7371.057942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7371.438818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7378.484532] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7386.544268] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7401.442247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7401.447557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7401.453223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7401.464653] Lustre: Skipped 2 previous similar messages [ 7403.769185] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.30@tcp (stopping) [ 7403.774929] Lustre: Skipped 2 previous similar messages [ 7406.336701] Lustre: server umount lustre-MDT0000 complete [ 7406.565931] LustreError: 321580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7406.577097] LustreError: 321580:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7408.857518] LustreError: 321576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7409.924281] LustreError: 322316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788276612 with bad export cookie 2539274692993585784 [ 7409.930089] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7409.943622] LustreError: 322316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7411.682953] LustreError: 321575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7411.684457] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7411.722314] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7414.011305] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.30@tcp (stopping) [ 7416.364672] Lustre: server umount lustre-MDT0001 complete [ 7422.689658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7431.082053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7435.233778] LustreError: 320890:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000071e402d3 x1875143842371840/t0(0) o250->MGC192.168.203.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7435.556363] LustreError: 323415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7435.580285] LustreError: 323415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7435.690169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7435.762740] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 7439.647679] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7444.904618] LustreError: 323416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7444.923170] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7444.940842] LustreError: 323416:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7449.080114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7449.629217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7454.733837] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7455.092274] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7465.115646] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7467.163616] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7472.922591] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7474.624227] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7480.312350] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7480.315676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7480.319088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7480.319096] Lustre: Skipped 1 previous similar message [ 7480.335243] Lustre: Skipped 1 previous similar message [ 7485.639599] Lustre: server umount lustre-OST0000 complete [ 7490.531553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7490.533097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7490.562690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7490.569524] Lustre: Skipped 4 previous similar messages [ 7495.577321] Lustre: server umount lustre-MDT0000 complete [ 7495.655579] LustreError: 325194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7495.676412] LustreError: 325194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7498.899489] LustreError: 321562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788276701 with bad export cookie 2539274692993593365 [ 7498.900373] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7498.920743] LustreError: 321562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7499.174715] Lustre: server umount lustre-MDT0001 complete [ 7509.655696] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 7512.580796] Key type lgssc unregistered [ 7512.886425] LNet: 327873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7512.896722] LNetError: 327873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7512.919574] LNet: Removed LNI 192.168.203.130@tcp [ 7513.978232] Key type .llcrypt unregistered [ 7513.980896] Key type ._llcrypt unregistered [ 7537.181990] Key type ._llcrypt registered [ 7537.184496] Key type .llcrypt registered [ 7537.330580] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 7538.796723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7538.920093] alg: No test for adler32 (adler32-zlib) [ 7540.052101] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 7540.311485] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 7542.003668] Key type lgssc registered [ 7543.608912] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7552.096878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7552.115732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7553.692619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7558.217541] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7564.268668] LustreError: 329291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7564.290441] LustreError: 329291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7567.261820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7567.812146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7572.449807] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7580.866749] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7586.497888] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7593.123231] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7593.152957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7593.155356] Lustre: Skipped 1 previous similar message [ 7596.486342] Lustre: server umount lustre-MDT0000 complete [ 7598.562552] LustreError: 330295:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7598.604682] LustreError: 330295:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7601.519187] LustreError: 329271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788276804 with bad export cookie 3228327258392266219 [ 7601.531779] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7601.535855] LustreError: 329271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7601.903073] Lustre: server umount lustre-MDT0001 complete [ 7615.805650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7616.042672] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7616.108714] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7616.334207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7621.164433] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7629.720455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7629.897515] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7629.945609] Lustre: 331878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7630.308311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7635.055390] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7641.775810] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7647.277860] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7656.878390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7657.098974] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 7657.120198] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 7657.140345] LustreError: 333760:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 7657.156550] LustreError: 333760:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7657.164195] LustreError: 333760:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7657.283897] Lustre: server umount lustre-OST0000 complete [ 7657.291894] LustreError: 333760:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7666.995145] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7667.172278] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7667.423704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7672.433906] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 7673.297335] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7680.361490] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7687.821965] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7687.928810] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7687.993862] Lustre: lustre-OST0001: new disk, initializing [ 7687.997184] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7688.002287] Lustre: 335301:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7688.082581] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7694.072457] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7694.984911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7694.997100] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7695.099729] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7701.533572] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7705.571148] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7705.576483] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7705.583480] Lustre: Skipped 2 previous similar messages [ 7705.603993] Lustre: Skipped 1 previous similar message [ 7710.618483] Lustre: server umount lustre-OST0001 complete [ 7710.705480] LustreError: 334226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7710.742845] LustreError: 334226:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7715.817628] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7715.820327] LustreError: 335724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7715.827967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7715.841598] Lustre: Skipped 1 previous similar message [ 7715.863627] LustreError: 335724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7720.567151] Lustre: server umount lustre-OST0000 complete [ 7726.050437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7726.071231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7726.099229] Lustre: Skipped 1 previous similar message [ 7726.101063] Lustre: Skipped 1 previous similar message [ 7730.652795] Lustre: server umount lustre-MDT0000 complete [ 7731.177909] LustreError: 331888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7731.197227] LustreError: 331888:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7734.242401] LustreError: 331869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788276936 with bad export cookie 3228327258392266772 [ 7734.243599] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7734.256524] LustreError: 331869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7734.941239] Lustre: server umount lustre-MDT0001 complete [ 7745.698599] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 7748.456861] Key type lgssc unregistered [ 7748.785875] LNet: 337263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7748.801249] LNetError: 337263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7748.817188] LNet: Removed LNI 192.168.203.130@tcp [ 7749.796854] Key type .llcrypt unregistered [ 7749.800275] Key type ._llcrypt unregistered [ 7820.395632] Key type ._llcrypt registered [ 7820.397384] Key type .llcrypt registered [ 7820.481281] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 7835.634978] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 7836.838313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7836.883092] alg: No test for adler32 (adler32-zlib) [ 7837.930737] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 7838.339269] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 7840.105100] Key type lgssc registered [ 7841.373481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7876.062568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7876.084687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7877.349926] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7877.376083] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7877.510554] Lustre: lustre-MDT0000: new disk, initializing [ 7877.606793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7877.627034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7882.474404] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7892.922294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7893.088628] Lustre: 342193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7893.134752] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7893.138909] Lustre: Skipped 1 previous similar message [ 7893.211988] Lustre: lustre-MDT0001: new disk, initializing [ 7893.302077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7893.342310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7893.354300] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7897.643479] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7906.821873] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7912.203340] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7918.876734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7919.125918] Lustre: lustre-OST0000: new disk, initializing [ 7919.129440] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7919.134853] Lustre: 344159:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7919.216899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7925.112232] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7927.893229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7927.907127] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7928.014763] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7933.163716] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7937.831949] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7938.023856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7941.972187] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7942.176248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7948.773279] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7948.775899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7948.780857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7948.798577] Lustre: Skipped 1 previous similar message [ 7948.848244] LustreError: Skipped 1 previous similar message [ 7950.231927] Lustre: server umount lustre-OST0000 complete [ 7954.914775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7954.941444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7954.970072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7954.989327] Lustre: Skipped 1 previous similar message [ 7959.012470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7959.019114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7959.046701] Lustre: Skipped 1 previous similar message [ 7964.136408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7964.142176] Lustre: Skipped 1 previous similar message [ 7969.248261] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7969.251149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7969.268049] Lustre: Skipped 1 previous similar message [ 7969.507249] Lustre: server umount lustre-MDT0000 complete [ 7973.185053] LustreError: 342186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788277175 with bad export cookie 11069520364470396797 [ 7973.187646] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7973.201683] LustreError: 342186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7973.832756] Lustre: server umount lustre-MDT0001 complete [ 7982.835334] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:39:44 (1788277184) [ 7990.622439] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:39:51 (1788277191) [ 7999.478666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8000.011661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8004.480583] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8005.097358] LustreError: 346916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8005.119892] LustreError: 346916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8010.214321] LustreError: 346915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8013.252933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8013.653035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8019.462427] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8028.248729] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8034.622584] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8042.239153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8042.775566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8049.821677] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8057.861589] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8065.804458] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 8068.014575] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8069.737811] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8071.902073] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 8074.249810] Lustre: Failing over lustre-MDT0000 [ 8074.728554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8074.734625] LustreError: 346912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8074.746341] Lustre: Skipped 1 previous similar message [ 8074.833849] Lustre: server umount lustre-MDT0000 complete [ 8079.851920] LustreError: 346916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8079.877341] LustreError: 346916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 8084.965607] LustreError: 346916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8084.996731] LustreError: 346916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8093.979145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8094.055417] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8094.321045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8094.372820] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8095.462575] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8099.223647] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8099.830428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8099.884896] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 8099.937299] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8108.638710] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8110.157742] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8113.842792] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 8115.446548] Lustre: Failing over lustre-MDT0000 [ 8115.732786] Lustre: server umount lustre-MDT0000 complete [ 8116.470399] LustreError: 348850:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8116.490967] LustreError: 348850:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 8120.296963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8120.298526] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8120.306691] Lustre: Skipped 3 previous similar messages [ 8132.909220] LustreError: 346912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8132.932776] LustreError: 346912:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 8135.648749] Lustre: 340419:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788277322/real 1788277322] req@0000000007f63049 x1875144385584128/t0(0) o400->MGC192.168.203.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788277338 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8135.700553] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8135.900150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8145.891781] LustreError: 340418:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000075d6fa54 x1875144385589120/t0(0) o250->MGC192.168.203.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8146.201129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8146.271494] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8147.162794] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8150.536959] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8151.537323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8151.545888] Lustre: Skipped 2 previous similar messages [ 8151.578196] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 8151.636333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8160.367568] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8161.940670] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8172.003207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8172.008268] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8172.021708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8172.027655] Lustre: Skipped 1 previous similar message [ 8172.043082] Lustre: Skipped 2 previous similar messages [ 8174.204399] Lustre: server umount lustre-MDT0000 complete [ 8177.123724] LustreError: 346916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8177.140466] LustreError: 346916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 8177.821320] LustreError: 346898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788277380 with bad export cookie 11069520364470399513 [ 8177.821619] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8177.835219] LustreError: 346898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8178.162954] Lustre: server umount lustre-MDT0001 complete [ 8192.481777] Lustre: server umount lustre-OST0000 complete [ 8207.389125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8212.340287] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8221.037164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8221.323485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8221.331830] Lustre: Skipped 1 previous similar message [ 8225.696315] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8232.884061] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8238.657489] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8245.407601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8252.160076] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8253.067226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8260.118651] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8267.403775] Lustre: server umount lustre-OST0000 complete [ 8272.353278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8272.359252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8272.366542] Lustre: Skipped 1 previous similar message [ 8272.372904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8281.195890] Lustre: server umount lustre-MDT0000 complete [ 8284.128062] Lustre: 340421:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788277470/real 1788277470] req@00000000a9c9cb06 x1875144385634688/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788277486 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8284.137937] LustreError: 353258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8284.212469] LustreError: 353258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 8284.882406] LustreError: 353239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788277487 with bad export cookie 11069520364470400255 [ 8284.883655] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8284.903251] LustreError: 353239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8285.239452] Lustre: server umount lustre-MDT0001 complete [ 8296.737208] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 8299.438717] Key type lgssc unregistered [ 8299.775941] LNet: 356790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8299.784437] LNetError: 356790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8299.813308] LNet: Removed LNI 192.168.203.130@tcp [ 8300.723223] Key type .llcrypt unregistered [ 8300.730791] Key type ._llcrypt unregistered [ 8331.974180] Key type ._llcrypt registered [ 8331.984327] Key type .llcrypt registered [ 8332.176311] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 8333.435646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8333.570359] alg: No test for adler32 (adler32-zlib) [ 8334.769766] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 8335.029151] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 8336.744147] Key type lgssc registered [ 8337.883724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8345.407729] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8345.428255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8346.866623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8350.711572] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8357.349766] LustreError: 358310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8357.364583] LustreError: 358310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8359.629344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8359.945746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8364.303303] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8373.887643] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8381.192491] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8389.129351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8389.406798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8392.507559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8396.113316] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8403.458851] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8415.202565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8415.216386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8415.234932] Lustre: Skipped 1 previous similar message [ 8416.265265] Lustre: server umount lustre-MDT0000 complete [ 8420.324620] LustreError: 359317:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8420.353292] LustreError: 359317:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8420.676951] LustreError: 360155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788277623 with bad export cookie 15917250254471919098 [ 8420.685096] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8420.688100] LustreError: 360155:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8421.198730] Lustre: server umount lustre-MDT0001 complete [ 8436.570763] Lustre: server umount lustre-OST0000 complete [ 8452.787605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8453.287653] LustreError: 362026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8453.344142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8458.112622] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8458.734175] LustreError: 362027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8462.822358] LustreError: 362026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8468.438413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8468.908601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8473.504319] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8481.478989] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8488.303862] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8496.361633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8496.861537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8497.904156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 8503.309935] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8510.581653] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8524.774888] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8524.791739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8524.799997] Lustre: Skipped 3 previous similar messages [ 8524.828779] Lustre: Skipped 4 previous similar messages [ 8527.607443] Lustre: server umount lustre-MDT0000 complete [ 8529.896076] LustreError: 362022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8529.937388] LustreError: 362022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8531.689571] LustreError: 362008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788277734 with bad export cookie 15917250254471920211 [ 8531.716873] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8532.153633] Lustre: server umount lustre-MDT0001 complete [ 8547.252834] Lustre: server umount lustre-OST0000 complete [ 8563.746128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8564.321839] LustreError: 365936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8564.400593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8569.452318] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8578.563987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8583.364262] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8590.810958] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8596.410247] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8602.705270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8603.110783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8603.128024] Lustre: Skipped 1 previous similar message [ 8609.296097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 8610.068771] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8618.138313] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8624.616479] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8624.625950] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8624.643532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8624.656911] Lustre: Skipped 1 previous similar message [ 8626.646165] Lustre: server umount lustre-OST0000 complete [ 8634.853566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8634.865372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8634.867105] Lustre: Skipped 2 previous similar messages [ 8634.886812] Lustre: Skipped 1 previous similar message [ 8637.175280] Lustre: server umount lustre-MDT0000 complete [ 8639.969985] LustreError: 365932:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8640.008307] LustreError: 365932:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8641.666030] LustreError: 365918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788277844 with bad export cookie 15917250254471921233 [ 8641.669937] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8641.673649] LustreError: 365918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8642.019795] Lustre: server umount lustre-MDT0001 complete [ 8654.186824] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 8658.616823] Key type lgssc unregistered [ 8659.108644] LNet: 369472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8659.141608] LNetError: 369472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8659.182649] LNet: Removed LNI 192.168.203.130@tcp [ 8660.329510] Key type .llcrypt unregistered [ 8660.335310] Key type ._llcrypt unregistered [ 8693.523562] Key type ._llcrypt registered [ 8693.526227] Key type .llcrypt registered [ 8693.659434] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 8694.693443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8694.728166] alg: No test for adler32 (adler32-zlib) [ 8695.909461] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 8696.243232] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 8697.904756] Key type lgssc registered [ 8699.160606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8707.491093] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8707.543315] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8707.556874] LustreError: 370914:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 8707.595312] LustreError: 370914:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 8707.601410] LustreError: 370914:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 8707.608208] LustreError: 370914:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 8707.615236] LustreError: 370914:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8715.224766] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8715.234963] LustreError: 371311:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 8715.259532] LustreError: 371311:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 8715.267841] LustreError: 371311:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 8715.275852] LustreError: 371311:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 8715.285526] LustreError: 371311:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8738.272399] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 8742.330882] Key type lgssc unregistered [ 8742.746939] LNet: 372319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8742.766961] LNetError: 372319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8742.807894] LNet: Removed LNI 192.168.203.130@tcp [ 8743.623582] Key type .llcrypt unregistered [ 8743.625315] Key type ._llcrypt unregistered [ 8763.970359] Key type ._llcrypt registered [ 8763.972642] Key type .llcrypt registered [ 8764.068618] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 8781.311122] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 8782.833861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8782.875704] alg: No test for adler32 (adler32-zlib) [ 8784.059753] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 8784.396257] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 8786.040754] Key type lgssc registered [ 8787.556407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8831.333264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8831.352539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8832.581667] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8832.616753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8832.725491] Lustre: lustre-MDT0000: new disk, initializing [ 8832.802563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8832.820950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8837.221983] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8848.073188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8848.266872] Lustre: 375479:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8848.357530] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8848.367174] Lustre: Skipped 1 previous similar message [ 8848.516578] Lustre: lustre-MDT0001: new disk, initializing [ 8848.649837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8848.704765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8848.723249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8853.514783] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8862.765340] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8867.971793] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8874.048156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8874.378648] Lustre: lustre-OST0000: new disk, initializing [ 8874.382419] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8874.389567] Lustre: 377445:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8874.473542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8880.698057] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8881.759379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8881.782376] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8881.943812] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8889.858577] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8895.802139] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8896.112322] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8901.887242] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8902.193526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8907.746528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8907.748155] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8907.760603] LustreError: Skipped 1 previous similar message [ 8907.761566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8907.805330] Lustre: Skipped 1 previous similar message [ 8910.570962] Lustre: server umount lustre-OST0000 complete [ 8915.428545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8915.441100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8915.458556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8915.465958] Lustre: Skipped 1 previous similar message [ 8917.987979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8917.990444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8921.432749] Lustre: server umount lustre-MDT0000 complete [ 8925.368425] LustreError: 376273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788278128 with bad export cookie 12437462598844498858 [ 8925.370731] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8925.383740] LustreError: 376273:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8925.926091] Lustre: server umount lustre-MDT0001 complete [ 8933.839641] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 8935.593536] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:55:37 (1788278137) [ 8942.846435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8943.389504] LustreError: 379902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8943.409329] LustreError: 379902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8943.489569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8948.052481] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8948.711505] LustreError: 379903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8953.826528] LustreError: 379903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8955.939860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8956.165600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8960.178152] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8967.453735] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8974.149807] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8981.965772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8982.409557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8990.958553] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8998.959755] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9005.893371] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9006.082785] Lustre: lustre-OST0001: new disk, initializing [ 9006.086430] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9006.092413] Lustre: 382741:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9006.193905] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9011.273153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9011.298265] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9011.390172] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9012.426559] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9022.401824] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9027.041628] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9027.048357] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9027.054085] LustreError: Skipped 1 previous similar message [ 9027.059865] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9027.062750] Lustre: Skipped 1 previous similar message [ 9027.069205] Lustre: Skipped 1 previous similar message [ 9032.173730] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9032.178747] Lustre: Skipped 1 previous similar message [ 9033.247390] Lustre: server umount lustre-OST0001 complete [ 9034.462216] LustreError: 383039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9034.493088] LustreError: 383039:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9042.408563] LustreError: 383039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9042.436110] LustreError: 383039:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9051.105816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9051.123287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9051.155799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9051.171289] Lustre: Skipped 1 previous similar message [ 9052.644679] LustreError: 383039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9052.685952] LustreError: 383039:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9056.917100] Lustre: server umount lustre-OST0000 complete [ 9062.882563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9062.904435] Lustre: Skipped 2 previous similar messages [ 9067.337240] Lustre: server umount lustre-MDT0000 complete [ 9071.085669] LustreError: 379884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788278273 with bad export cookie 12437462598844499733 [ 9071.086489] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9071.101186] LustreError: 379884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9077.756210] Lustre: server umount lustre-MDT0001 complete [ 9089.275350] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 9092.576304] Key type lgssc unregistered [ 9092.886346] LNet: 384732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9092.896795] LNetError: 384732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9092.913082] LNet: Removed LNI 192.168.203.130@tcp [ 9093.738525] Key type .llcrypt unregistered [ 9093.743922] Key type ._llcrypt unregistered [ 9144.505267] Key type ._llcrypt registered [ 9144.513370] Key type .llcrypt registered [ 9144.652197] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 9145.829544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9145.902081] alg: No test for adler32 (adler32-zlib) [ 9146.972662] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 9147.288417] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 9149.002559] Key type lgssc registered [ 9150.229773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9157.559715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9157.596236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9158.854906] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9158.913316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9159.089293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9163.008855] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9171.381258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9171.488042] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9171.538244] Lustre: 387533:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9171.764244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9176.266384] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9183.414554] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9189.942851] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9192.418919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9192.429374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9192.446396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9192.930157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9192.963096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9198.050221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9198.058657] Lustre: Skipped 1 previous similar message [ 9198.315924] Lustre: server umount lustre-MDT0000 complete [ 9202.193649] LustreError: 387524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788278404 with bad export cookie 11243415672858633204 [ 9202.200942] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9202.210318] LustreError: 387524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9202.552956] Lustre: server umount lustre-MDT0001 complete [ 9210.898939] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:00:12 (1788278412) [ 9217.756212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9218.298543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9223.187922] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9223.649513] LustreError: 390044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9223.672375] LustreError: 390044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9228.770259] LustreError: 390045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9231.710939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9232.055976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9236.215874] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9242.966555] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9248.730467] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9255.465364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9255.744574] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9256.036066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9262.730435] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9270.278843] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9278.699913] Lustre: Permanently deactivating lustre-OST0000 [ 9278.720082] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 9285.165893] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9286.430185] LustreError: 392991:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 9286.441587] LustreError: 392991:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 9286.449266] LustreError: 392991:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9289.384546] LustreError: 393039:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 9289.389849] LustreError: 393039:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9297.375530] Lustre: server umount lustre-OST0000 complete [ 9300.419109] LustreError: 393289:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 9300.429776] LustreError: 393289:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9302.502471] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9302.518081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9302.520951] Lustre: Skipped 1 previous similar message [ 9307.618375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9307.630430] Lustre: Skipped 2 previous similar messages [ 9311.786064] Lustre: server umount lustre-MDT0000 complete [ 9312.751986] LustreError: 391104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9312.779247] LustreError: 391104:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9315.921250] LustreError: 390024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788278518 with bad export cookie 11243415672858633785 [ 9315.926277] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9315.941534] LustreError: 390024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9325.800692] Lustre: server umount lustre-MDT0001 complete [ 9334.536952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9338.760704] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9341.161755] LustreError: 394451:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 9346.561880] Lustre: 394789:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.130@tcp [ 9348.444640] Lustre: server umount MGS complete [ 9358.505923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9358.869371] LustreError: 395513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9358.887942] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9358.899916] Lustre: Skipped 1 previous similar message [ 9358.956296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9362.568426] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9364.454217] LustreError: 395514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9370.063790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9370.237680] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9373.987390] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9380.841547] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9387.084879] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9388.745190] Lustre: Permanently reactivating lustre-OST0000 [ 9388.766193] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9388.774174] Lustre: Skipped 2 previous similar messages [ 9415.137066] LustreError: 397251:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 9422.203459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9422.467418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9422.471902] Lustre: Skipped 1 previous similar message [ 9427.822130] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9434.396444] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9440.229190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9440.236159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9440.238189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9440.245363] LustreError: Skipped 1 previous similar message [ 9440.274718] Lustre: Skipped 1 previous similar message [ 9445.714364] Lustre: server umount lustre-OST0000 complete [ 9450.471742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9456.755938] Lustre: server umount lustre-MDT0000 complete [ 9460.706772] LustreError: 397153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9460.755058] LustreError: 397153:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9460.814184] LustreError: 395495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788278663 with bad export cookie 11243415672858634835 [ 9460.835956] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9461.542921] Lustre: server umount lustre-MDT0001 complete [ 9473.790338] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 9476.546162] Key type lgssc unregistered [ 9476.834338] LNet: 399929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9476.856530] LNetError: 399929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9477.928856] LNet: Removed LNI 192.168.203.130@tcp [ 9479.006288] Key type .llcrypt unregistered [ 9479.009455] Key type ._llcrypt unregistered [ 9497.269582] Key type ._llcrypt registered [ 9497.272541] Key type .llcrypt registered [ 9497.375474] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 9511.710527] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 9512.580841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9512.689361] alg: No test for adler32 (adler32-zlib) [ 9513.889760] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 9514.167023] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 9515.872312] Key type lgssc registered [ 9516.940667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9548.083824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9548.111754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9549.431809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9549.466959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9549.582181] Lustre: lustre-MDT0000: new disk, initializing [ 9549.670191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9549.694695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9554.417790] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9565.401675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9565.513088] Lustre: 403083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9565.591299] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9565.598374] Lustre: Skipped 1 previous similar message [ 9565.601571] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9565.732848] Lustre: lustre-MDT0001: new disk, initializing [ 9565.891506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9565.936767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9565.943989] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9570.166794] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9579.752425] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9585.983039] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9595.108857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9595.562549] Lustre: lustre-OST0000: new disk, initializing [ 9595.570398] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9595.578921] Lustre: 405051:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9595.698047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9602.966709] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9605.689330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9605.713637] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9605.802031] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9612.532225] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9617.932519] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9618.102399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9622.277376] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9622.479587] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9626.596204] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9626.598214] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9626.609608] LustreError: Skipped 1 previous similar message [ 9626.610963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9626.623991] Lustre: Skipped 1 previous similar message [ 9626.639413] Lustre: Skipped 1 previous similar message [ 9630.428679] Lustre: server umount lustre-OST0000 complete [ 9636.834616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9636.842540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9636.847574] Lustre: Skipped 1 previous similar message [ 9640.730900] Lustre: server umount lustre-MDT0000 complete [ 9644.647668] LustreError: 403075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788278847 with bad export cookie 7495899813726670240 [ 9644.654223] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9644.660839] LustreError: 403075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9644.978771] Lustre: server umount lustre-MDT0001 complete [ 9653.303821] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 9654.892536] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:07:36 (1788278856) [ 9661.614278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9662.201343] LustreError: 407511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9662.233793] LustreError: 407511:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9662.303830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9667.063466] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9667.556037] LustreError: 407512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9671.656615] LustreError: 407511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9674.809558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9675.223674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9679.688328] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9687.840746] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9694.093979] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9702.329711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9702.693814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9709.677928] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9716.822977] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9718.679404] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 9721.184920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 9727.459284] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9727.469153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9727.487974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9727.493323] Lustre: Skipped 1 previous similar message [ 9730.536787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9730.558282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9732.793467] Lustre: server umount lustre-OST0000 complete [ 9736.673774] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9736.688876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9736.710201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9736.718884] Lustre: Skipped 2 previous similar messages [ 9742.393725] Lustre: server umount lustre-MDT0000 complete [ 9745.890245] LustreError: 408520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9745.917956] LustreError: 408520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9746.032251] LustreError: 407493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788278948 with bad export cookie 7495899813726671122 [ 9746.043404] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9746.052601] LustreError: 407493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9746.525250] Lustre: server umount lustre-MDT0001 complete [ 9757.913666] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 9760.617903] Key type lgssc unregistered [ 9760.973578] LNet: 411243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9760.998469] LNetError: 411243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9761.036116] LNet: Removed LNI 192.168.203.130@tcp [ 9762.118267] Key type .llcrypt unregistered [ 9762.120605] Key type ._llcrypt unregistered [ 9787.906831] Key type ._llcrypt registered [ 9787.909375] Key type .llcrypt registered [ 9788.021957] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 9788.480654] Key type .llcrypt unregistered [ 9788.490633] Key type ._llcrypt unregistered [ 9809.586414] Key type ._llcrypt registered [ 9809.594160] Key type .llcrypt registered [ 9809.735928] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 9811.076641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9811.218205] alg: No test for adler32 (adler32-zlib) [ 9812.280891] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 9812.548209] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 9814.232254] Key type lgssc registered [ 9815.446068] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9823.040715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9823.070787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9824.694585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9828.666675] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9834.978526] LustreError: 413461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9835.008290] LustreError: 413461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9837.201723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9837.754958] LustreError: 413462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9837.791759] LustreError: 413462:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9837.798557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9843.172509] LustreError: 413458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9843.179149] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9843.185453] LustreError: 413458:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9848.290569] LustreError: 413457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9848.304861] LustreError: 413457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9850.315926] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9850.369604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9855.464629] LustreError: 413461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9855.477637] LustreError: 413461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9857.572662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9857.852863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9862.976162] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9873.277659] Lustre: server umount lustre-OST0000 complete [ 9878.498779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9878.519174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9878.519586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9878.548728] Lustre: Skipped 1 previous similar message [ 9883.624945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9883.634375] Lustre: Skipped 1 previous similar message [ 9886.827162] Lustre: server umount lustre-MDT0000 complete [ 9888.738973] LustreError: 413461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9888.748150] LustreError: 413461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9889.760368] Lustre: 412772:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788279076/real 1788279076] req@00000000c0bbf4b5 x1875146456459264/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788279092 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9889.763178] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9889.788737] Lustre: 412772:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 9889.845511] Lustre: Skipped 1 previous similar message [ 9890.736884] LustreError: 413442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788279093 with bad export cookie 2499304990033965613 [ 9890.737598] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9890.746727] LustreError: 413442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9891.116776] Lustre: server umount lustre-MDT0001 complete [ 9900.827326] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 9903.542047] Key type lgssc unregistered [ 9903.802984] LNet: 416611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9903.813274] LNetError: 416611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9903.832523] LNet: Removed LNI 192.168.203.130@tcp [ 9904.602886] Key type .llcrypt unregistered [ 9904.607106] Key type ._llcrypt unregistered [ 9926.651619] Key type ._llcrypt registered [ 9926.655422] Key type .llcrypt registered [ 9926.755878] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 9927.907227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9927.925234] alg: No test for adler32 (adler32-zlib) [ 9929.094129] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [ 9929.374959] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 9931.056308] Key type lgssc registered [ 9932.258566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9940.618449] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9940.639606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9956.256913] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 9961.376328] LustreError: 418003:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.130@tcp: failed processing log, type 4: rc = -110 [ 9988.128213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9994.437508] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10001.330437] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10007.389200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10007.940253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10008.033932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10012.494742] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10013.178466] LustreError: 418028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10013.198679] LustreError: 418028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10018.275137] LustreError: 418027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10020.299024] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10020.576552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10024.391824] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10030.676487] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10036.310810] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10043.875457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10043.886670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10043.909679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10044.393344] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10044.411058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10044.417266] Lustre: Skipped 1 previous similar message [10048.660799] Lustre: server umount lustre-OST0000 complete [10054.629180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10054.640828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10054.647930] Lustre: Skipped 1 previous similar message [10058.160116] Lustre: server umount lustre-MDT0000 complete [10059.746368] LustreError: 419044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10059.759481] LustreError: 419044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10061.361911] LustreError: 418009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788279264 with bad export cookie 2958805389092316977 [10061.369741] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10061.392136] LustreError: 418009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10062.128986] Lustre: server umount lustre-MDT0001 complete [10073.326636] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [10076.283625] Key type lgssc unregistered [10076.532853] LNet: 421577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10076.571078] LNetError: 421577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10076.600042] LNet: Removed LNI 192.168.203.130@tcp [10077.379945] Key type .llcrypt unregistered [10077.382688] Key type ._llcrypt unregistered [10101.513129] Key type ._llcrypt registered [10101.517240] Key type .llcrypt registered [10101.662522] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [10102.927380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10102.957668] alg: No test for adler32 (adler32-zlib) [10103.978302] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [10104.193648] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [10105.875258] Key type lgssc registered [10107.007171] Lustre: Echo OBD driver; http://www.lustre.org/ [10115.260263] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10115.310226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10116.708716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10121.486742] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10126.819568] LustreError: 422996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10126.833598] LustreError: 422996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10130.016704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10130.324684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10134.627478] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10142.653077] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10147.771259] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10154.116862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10154.478727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10160.651220] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10161.663474] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10167.886862] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10171.877458] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10171.881987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10171.903486] Lustre: Skipped 1 previous similar message [10175.210658] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.30@tcp (stopping) [10175.220122] Lustre: Skipped 1 previous similar message [10176.492680] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10176.498635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10176.508493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10177.838387] Lustre: server umount lustre-MDT0000 complete [10180.712741] LustreError: 422976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.30@tcp arrived at 1788279383 with bad export cookie 3788203770450074737 [10180.734337] LustreError: 422976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10182.118415] LustreError: 422996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10182.140126] LustreError: 422996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10183.124466] LustreError: 422977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788279385 with bad export cookie 3788203770450073841 [10183.143164] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10186.211218] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10186.220570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10186.235046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10186.245306] Lustre: Skipped 2 previous similar messages [10187.233689] LustreError: 423724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10187.245306] LustreError: 423724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10189.484231] Lustre: server umount lustre-OST0000 complete [10192.355020] LustreError: 423724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10192.382439] LustreError: 423724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10196.677708] Lustre: server umount lustre-MDT0001 complete [10207.241718] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [10210.013327] Key type lgssc unregistered [10210.331245] LNet: 426682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10210.335837] LNetError: 426682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10211.369867] LNet: Removed LNI 192.168.203.130@tcp [10212.212207] Key type .llcrypt unregistered [10212.215521] Key type ._llcrypt unregistered [10234.506344] Key type ._llcrypt registered [10234.513629] Key type .llcrypt registered [10234.661440] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [10236.104991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10236.134064] alg: No test for adler32 (adler32-zlib) [10237.268462] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [10237.534553] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [10239.256193] Key type lgssc registered [10240.290477] Lustre: Echo OBD driver; http://www.lustre.org/ [10247.646319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10247.659701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10249.311466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10254.407741] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10258.403996] LustreError: 428099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10258.422795] LustreError: 428099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10263.532559] LustreError: 428098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10263.652621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10264.005518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10268.593683] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10276.871759] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10283.240214] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10291.735572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10292.209753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10294.272471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10300.024885] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10308.592020] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10313.708060] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10313.743946] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10313.757920] Lustre: Skipped 1 previous similar message [10314.917798] Lustre: server umount lustre-MDT0001 complete [10315.520078] LustreError: 428094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10318.817697] LustreError: 428098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10318.856555] LustreError: 428098:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10323.938646] LustreError: 429106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10323.938774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10323.954394] Lustre: Skipped 1 previous similar message [10323.958873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10327.874196] Lustre: server umount lustre-OST0000 complete [10332.766644] Lustre: server umount lustre-MDT0000 complete [10350.391210] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [10354.175255] Key type lgssc unregistered [10354.552710] LNet: 431789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10354.566854] LNetError: 431789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10355.624303] LNet: Removed LNI 192.168.203.130@tcp [10356.712431] Key type .llcrypt unregistered [10356.716333] Key type ._llcrypt unregistered [10381.005867] Key type ._llcrypt registered [10381.010188] Key type .llcrypt registered [10381.157452] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [10397.815553] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [10398.682014] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10398.709916] alg: No test for adler32 (adler32-zlib) [10399.839384] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [10400.138518] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [10401.816194] Key type lgssc registered [10402.977790] Lustre: Echo OBD driver; http://www.lustre.org/ [10443.295091] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [10458.583081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10458.625227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10459.952608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10459.995232] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10460.060424] Lustre: lustre-MDT0000: new disk, initializing [10460.128414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10460.156268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10464.643462] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10479.628462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10479.884364] Lustre: lustre-OST0000: new disk, initializing [10479.890261] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10479.894943] Lustre: Skipped 1 previous similar message [10479.902957] Lustre: 436830:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10479.988152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10485.390474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10485.405047] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10485.446122] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10486.016480] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10495.471694] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10509.862299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10510.020075] Lustre: 435814:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10510.100637] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10510.191429] Lustre: lustre-MDT0001: new disk, initializing [10510.314086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10510.367358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [10510.383966] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [10514.724770] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10523.484522] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [10523.658940] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [10528.412948] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [10528.648848] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [10537.455965] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10537.456436] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10537.483825] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10537.489622] Lustre: Skipped 1 previous similar message [10539.708837] Lustre: server umount lustre-MDT0001 complete [10547.682747] LustreError: 435825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10547.701723] LustreError: 435825:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [10552.807570] LustreError: 435824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10552.827555] LustreError: 435824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10557.410092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10557.422112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10557.438375] Lustre: Skipped 1 previous similar message [10557.443769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10557.449663] Lustre: Skipped 1 previous similar message [10557.922709] LustreError: 435825:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10563.042263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10563.042687] LustreError: 435824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10563.434945] Lustre: server umount lustre-OST0000 complete [10573.780658] Lustre: server umount lustre-MDT0000 complete [10587.448904] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [10590.593893] Key type lgssc unregistered [10591.023182] LNet: 440575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10591.042946] LNetError: 440575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10591.066943] LNet: Removed LNI 192.168.203.130@tcp [10592.045984] Key type .llcrypt unregistered [10592.051077] Key type ._llcrypt unregistered [10629.949766] Key type ._llcrypt registered [10629.962030] Key type .llcrypt registered [10630.126682] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [10630.587083] Key type .llcrypt unregistered [10630.597159] Key type ._llcrypt unregistered [10646.113570] Key type ._llcrypt registered [10646.116852] Key type .llcrypt registered [10646.191258] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [10647.074217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10647.205588] alg: No test for adler32 (adler32-zlib) [10648.288371] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [10648.568659] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [10650.216154] Key type lgssc registered [10651.479644] Lustre: Echo OBD driver; http://www.lustre.org/ [10678.408325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10678.433176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10679.809958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10679.864656] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10679.940408] Lustre: lustre-MDT0000: new disk, initializing [10680.003626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10680.019418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10684.367339] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10694.251346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10694.376980] Lustre: 443955:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10694.420337] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10694.424306] Lustre: Skipped 1 previous similar message [10694.520625] Lustre: lustre-MDT0001: new disk, initializing [10694.557712] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10694.645518] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10694.682522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10694.694489] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10698.781805] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10707.628054] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10712.235354] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10718.963329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10719.165617] Lustre: lustre-OST0000: new disk, initializing [10719.169847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10719.174641] Lustre: 445926:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10719.249630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10725.280941] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10728.052131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10728.057585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10728.178218] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10734.432649] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10740.707730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10740.715708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10740.732506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10743.779921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10743.784609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10747.021982] Lustre: server umount lustre-MDT0000 complete [10750.530720] LustreError: 443949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788279953 with bad export cookie 14003590571673334446 [10750.538770] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10750.543730] LustreError: 443949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10750.875407] Lustre: server umount lustre-MDT0001 complete [10764.511810] Lustre: server umount lustre-OST0000 complete [10773.941407] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:26:15 (1788279975) [10780.924650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10781.272206] LustreError: 447954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10781.302255] LustreError: 447954:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10781.364738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10785.169824] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10786.792578] LustreError: 447955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10790.883132] LustreError: 447954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10792.418666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10792.770748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10796.824335] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10802.847635] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10807.183843] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10826.041738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10826.311309] Lustre: Found index 0 for lustre-OST0000, updating log [10826.500805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10834.095842] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10842.513472] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10852.095358] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [10862.570809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10862.584523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10862.618521] Lustre: Skipped 1 previous similar message [10862.628690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10862.638816] Lustre: Skipped 2 previous similar messages [10866.839234] Lustre: server umount lustre-OST0000 complete [10885.096580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10885.101930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10885.104485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10885.104493] Lustre: Skipped 3 previous similar messages [10885.168703] Lustre: Skipped 2 previous similar messages [10886.176296] LNet: 1 local NIs in recovery (showing 1): 192.168.203.130@tcp [10888.690892] Lustre: server umount lustre-MDT0000 complete [10890.216888] LustreError: 447954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10890.234296] LustreError: 447954:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10892.129022] LustreError: 447934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788280094 with bad export cookie 14003590571673335839 [10892.141875] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10892.149732] LustreError: 447934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10892.489274] Lustre: server umount lustre-MDT0001 complete [10899.715526] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:28:21 (1788280101) [10949.664448] LNet: 1 local NIs in recovery (showing 1): 192.168.203.130@tcp [11511.413918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11511.749596] LustreError: 454382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11511.777625] LustreError: 454382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11511.818523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11515.309490] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11516.898821] LustreError: 454383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11522.039593] LustreError: 454382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11523.099364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11523.505316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11527.959166] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11534.537869] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11539.162923] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11558.705510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11558.947697] Lustre: Found index 0 for lustre-OST0000, updating log [11559.304281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11560.304964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [11564.599838] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11565.088379] LNet: 1 local NIs in recovery (showing 1): 192.168.203.130@tcp [11571.687245] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11574.888817] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11574.916692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11574.933878] Lustre: Skipped 1 previous similar message [11580.201588] Lustre: server umount lustre-OST0000 complete [11584.935894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11584.948879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11584.955529] Lustre: Skipped 1 previous similar message [11586.983140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11586.997747] Lustre: Skipped 3 previous similar messages [11591.076505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11591.086845] Lustre: Skipped 7 previous similar messages [11598.304141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11598.524212] Lustre: server umount lustre-MDT0000 complete [11599.266594] LustreError: 454377:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11599.288459] LustreError: 454377:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11602.076450] LustreError: 454364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788280804 with bad export cookie 14003590571673336924 [11602.085788] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11602.088591] LustreError: 454364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11602.494444] Lustre: server umount lustre-MDT0001 complete [11628.640470] LNet: 1 local NIs in recovery (showing 1): 192.168.203.130@tcp [11634.024691] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [11642.019080] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [11678.613427] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:41:19 (1788280879) [11700.665371] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [11703.936689] Key type lgssc unregistered [11704.183687] LNet: 461001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11704.193301] LNetError: 461001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11823.016447] LNet: Removed LNI 192.168.203.130@tcp [11824.035369] Key type .llcrypt unregistered [11824.041783] Key type ._llcrypt unregistered [11840.725833] Key type ._llcrypt registered [11840.727941] Key type .llcrypt registered [11840.818340] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [11842.175425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11842.198982] alg: No test for adler32 (adler32-zlib) [11843.278406] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [11843.497263] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [11845.160992] Key type lgssc registered [11846.393916] Lustre: Echo OBD driver; http://www.lustre.org/ [11850.725810] LNet: Added LNI 192.168.203.130@tcp100 [8/256/0/180] [11851.411664] LNet: Added LNI 192.168.203.130@tcp101 [8/256/0/180] [11852.196133] LNet: Added LNI 192.168.203.130@tcp102 [8/256/0/180] [11853.010963] LNet: Added LNI 192.168.203.130@tcp103 [8/256/0/180] [11854.046849] LNet: Added LNI 192.168.203.130@tcp104 [8/256/0/180] [11854.903984] LNet: Added LNI 192.168.203.130@tcp105 [8/256/0/180] [11855.801853] LNet: Added LNI 192.168.203.130@tcp106 [8/256/0/180] [11856.799996] LNet: Added LNI 192.168.203.130@tcp107 [8/256/0/180] [11857.676324] LNet: Added LNI 192.168.203.130@tcp108 [8/256/0/180] [11858.499585] LNet: Added LNI 192.168.203.130@tcp109 [8/256/0/180] [11859.427571] LNet: Added LNI 192.168.203.130@tcp110 [8/256/0/180] [11860.203447] LNet: Added LNI 192.168.203.130@tcp111 [8/256/0/180] [11861.491150] LNet: Added LNI 192.168.203.130@tcp112 [8/256/0/180] [11862.532685] LNet: Added LNI 192.168.203.130@tcp113 [8/256/0/180] [11863.516574] LNet: Added LNI 192.168.203.130@tcp114 [8/256/0/180] [11864.456285] LNet: Added LNI 192.168.203.130@tcp115 [8/256/0/180] [11865.303781] LNet: Added LNI 192.168.203.130@tcp116 [8/256/0/180] [11866.221420] LNet: Added LNI 192.168.203.130@tcp117 [8/256/0/180] [11866.990906] LNet: Added LNI 192.168.203.130@tcp118 [8/256/0/180] [11867.832420] LNet: Added LNI 192.168.203.130@tcp119 [8/256/0/180] [11868.569162] LNet: Added LNI 192.168.203.130@tcp120 [8/256/0/180] [11869.441911] LNet: Added LNI 192.168.203.130@tcp121 [8/256/0/180] [11870.283448] LNet: Added LNI 192.168.203.130@tcp122 [8/256/0/180] [11871.053583] LNet: Added LNI 192.168.203.130@tcp123 [8/256/0/180] [11871.893993] LNet: Added LNI 192.168.203.130@tcp124 [8/256/0/180] [11872.840730] LNet: Added LNI 192.168.203.130@tcp125 [8/256/0/180] [11873.570860] LNet: Added LNI 192.168.203.130@tcp126 [8/256/0/180] [11874.288316] LNet: Added LNI 192.168.203.130@tcp127 [8/256/0/180] [11875.103311] LNet: Added LNI 192.168.203.130@tcp128 [8/256/0/180] [11875.952508] LNet: Added LNI 192.168.203.130@tcp129 [8/256/0/180] [11876.787992] LNet: Added LNI 192.168.203.130@tcp130 [8/256/0/180] [11877.511206] LNet: Added LNI 192.168.203.130@tcp131 [8/256/0/180] [11878.459972] LNet: Added LNI 192.168.203.130@tcp132 [8/256/0/180] [11879.450611] LNet: Added LNI 192.168.203.130@tcp133 [8/256/0/180] [11880.398843] LNet: Added LNI 192.168.203.130@tcp134 [8/256/0/180] [11881.247792] LNet: Added LNI 192.168.203.130@tcp135 [8/256/0/180] [11887.839167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11887.858586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11888.988176] Lustre: 464034:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.130@tcp: can't add NID 192.168.203.130@tcp130: rc = -75 [11889.034812] LNetError: 464057:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [11889.060412] LNetError: Protocol error connecting to 192.168.203.130@tcp118 at host 192.168.203.130:988: is it running a compatible version of Lustre? [11889.076037] LNetError: Skipped 4 previous similar messages [11889.215274] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11889.251970] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11889.360429] Lustre: lustre-MDT0000: new disk, initializing [11889.437184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11889.465075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11890.083206] LNetError: 461496:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [11890.099931] LNetError: 461496:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [11890.105958] LNetError: Protocol error connecting to 192.168.203.130@tcp111 at host 192.168.203.130:988: is it running a compatible version of Lustre? [11893.459705] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11902.992666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11903.145671] Lustre: 464050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11903.250778] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11903.257208] Lustre: Skipped 1 previous similar message [11903.351588] Lustre: lustre-MDT0001: new disk, initializing [11903.401068] Lustre: 464852:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.130@tcp: can't add NID 192.168.203.130@tcp130: rc = -75 [11903.412051] Lustre: 464852:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11903.542824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11903.583305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11903.598063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11906.668885] Lustre: 465050:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.130@tcp: can't add NID 192.168.203.130@tcp130: rc = -75 [11906.679065] Lustre: 465050:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11907.632394] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11916.218458] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11919.269159] LNetError: 464065:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [11919.279444] LNetError: Protocol error connecting to 192.168.203.130@tcp104 at host 192.168.203.130:988: is it running a compatible version of Lustre? [11921.650884] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11922.337541] LNetError: 464060:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [11922.350194] LNetError: Protocol error connecting to 192.168.203.130@tcp102 at host 192.168.203.130:988: is it running a compatible version of Lustre? [11927.465890] LNetError: 461496:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [11927.492281] LNetError: 461496:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [11927.499358] LNetError: Protocol error connecting to 192.168.203.130@tcp103 at host 192.168.203.130:988: is it running a compatible version of Lustre? [11927.511357] LNetError: Skipped 3 previous similar messages [11927.868867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11928.127963] Lustre: lustre-OST0000: new disk, initializing [11928.136723] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11928.150711] Lustre: 466026:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11928.247341] Lustre: 466031:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.130@tcp: can't add NID 192.168.203.130@tcp130: rc = -75 [11928.265236] Lustre: 466031:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [11928.318801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11933.754629] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11935.651957] LNetError: 461494:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [11935.671875] LNetError: 461494:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [11935.684438] LNetError: Protocol error connecting to 192.168.203.130@tcp102 at host 192.168.203.130:988: is it running a compatible version of Lustre? [11935.695034] LNetError: Skipped 3 previous similar messages [11936.265062] Lustre: 466483:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.130@tcp: can't add NID 192.168.203.130@tcp130: rc = -75 [11936.271605] Lustre: 466483:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [11936.327398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11936.348589] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11936.449510] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11941.347196] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11946.990128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11946.996301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11947.001736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11947.010250] Lustre: Skipped 1 previous similar message [11952.104158] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11952.109186] Lustre: Skipped 2 previous similar messages [11953.061371] LNetError: 464066:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [11953.074680] LNetError: 464066:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [11953.079627] LNetError: Protocol error connecting to 192.168.203.130@tcp111 at host 192.168.203.130:988: is it running a compatible version of Lustre? [11953.089142] LNetError: Skipped 9 previous similar messages [11957.220902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11957.225940] Lustre: Skipped 1 previous similar message [11961.321181] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11961.492407] Lustre: server umount lustre-OST0000 complete [11967.457480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11967.491670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11967.499643] Lustre: Skipped 1 previous similar message [11971.382514] Lustre: server umount lustre-MDT0000 complete [11974.578136] LustreError: 464043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788281177 with bad export cookie 9018199201902742017 [11974.585910] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11974.588454] LustreError: 464043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11974.859158] Lustre: server umount lustre-MDT0001 complete [11985.525333] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [11988.336975] Key type lgssc unregistered [11988.761088] LNet: 467943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11988.767679] LNetError: 467943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11988.771619] LNet: Removed LNI 192.168.203.130@tcp [11989.792259] LNet: Removed LNI 192.168.203.130@tcp100 [11989.796331] LNetError: 467943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11989.806815] LNetError: 467943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11989.811902] LNet: Removed LNI 192.168.203.130@tcp101 [11990.816130] LNet: Removed LNI 192.168.203.130@tcp102 [11990.822925] LNetError: 467943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11990.840138] LNetError: 467943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11990.846209] LNet: Removed LNI 192.168.203.130@tcp103 [11991.904213] LNet: Removed LNI 192.168.203.130@tcp104 [11992.928337] LNet: Removed LNI 192.168.203.130@tcp105 [11992.935677] LNetError: 467943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11992.950436] LNetError: 467943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [11992.963932] LNet: Removed LNI 192.168.203.130@tcp106 [11994.016728] LNet: Removed LNI 192.168.203.130@tcp107 [11995.040307] LNet: Removed LNI 192.168.203.130@tcp108 [11996.064185] LNet: Removed LNI 192.168.203.130@tcp109 [11997.088648] LNet: Removed LNI 192.168.203.130@tcp110 [11997.095709] LNetError: 467943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11997.101308] LNetError: 467943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [11997.105665] LNet: Removed LNI 192.168.203.130@tcp111 [11998.112182] LNet: Removed LNI 192.168.203.130@tcp112 [11999.136096] LNet: Removed LNI 192.168.203.130@tcp113 [12000.161094] LNet: Removed LNI 192.168.203.130@tcp114 [12001.184178] LNet: Removed LNI 192.168.203.130@tcp115 [12002.208181] LNet: Removed LNI 192.168.203.130@tcp116 [12003.233457] LNet: Removed LNI 192.168.203.130@tcp117 [12004.256192] LNet: Removed LNI 192.168.203.130@tcp118 [12005.280685] LNet: Removed LNI 192.168.203.130@tcp119 [12005.283846] LNetError: 467943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12005.296249] LNetError: 467943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [12005.311568] LNet: Removed LNI 192.168.203.130@tcp120 [12006.368289] LNet: Removed LNI 192.168.203.130@tcp121 [12007.393891] LNet: Removed LNI 192.168.203.130@tcp122 [12008.416192] LNet: Removed LNI 192.168.203.130@tcp123 [12009.440438] LNet: Removed LNI 192.168.203.130@tcp124 [12010.466228] LNet: Removed LNI 192.168.203.130@tcp125 [12011.488189] LNet: Removed LNI 192.168.203.130@tcp126 [12012.516507] LNet: Removed LNI 192.168.203.130@tcp127 [12013.537210] LNet: Removed LNI 192.168.203.130@tcp128 [12013.540666] LNet: Removed LNI 192.168.203.130@tcp129 [12013.549757] LNet: Removed LNI 192.168.203.130@tcp130 [12013.558290] LNet: Removed LNI 192.168.203.130@tcp131 [12013.564720] LNet: Removed LNI 192.168.203.130@tcp132 [12013.569780] LNet: Removed LNI 192.168.203.130@tcp133 [12013.573753] LNet: Removed LNI 192.168.203.130@tcp134 [12013.600256] LNet: Removed LNI 192.168.203.130@tcp135 [12014.446090] Key type .llcrypt unregistered [12014.448127] Key type ._llcrypt unregistered [12037.233792] Key type ._llcrypt registered [12037.236149] Key type .llcrypt registered [12037.310544] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [12051.601910] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [12052.905727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12053.001792] alg: No test for adler32 (adler32-zlib) [12054.087933] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [12054.434168] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [12056.128306] Key type lgssc registered [12057.382709] Lustre: Echo OBD driver; http://www.lustre.org/ [12093.378682] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:48:15 (1788281295) [12110.384612] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [12113.001199] Key type lgssc unregistered [12113.286713] LNet: 472034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12113.292750] LNetError: 472034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12113.309276] LNet: Removed LNI 192.168.203.130@tcp [12114.138450] Key type .llcrypt unregistered [12114.141090] Key type ._llcrypt unregistered [12131.360376] Key type ._llcrypt registered [12131.366759] Key type .llcrypt registered [12131.508364] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [12132.800401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12132.842382] alg: No test for adler32 (adler32-zlib) [12134.069565] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [12134.419684] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [12136.152349] Key type lgssc registered [12137.310158] Lustre: Echo OBD driver; http://www.lustre.org/ [12146.640994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12146.700832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12148.015706] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12148.042118] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12148.124908] Lustre: lustre-MDT0000: new disk, initializing [12148.262206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12148.281949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12152.925571] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12165.386162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12165.559451] Lustre: 473245:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12165.640948] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12165.647893] Lustre: Skipped 1 previous similar message [12165.751046] Lustre: lustre-MDT0001: new disk, initializing [12165.888443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12165.944646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12165.961691] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12170.954084] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12181.774720] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12187.550596] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12195.253086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12195.756854] Lustre: lustre-OST0000: new disk, initializing [12195.770457] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12195.786373] Lustre: 475220:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12195.957974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12201.617418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12201.629629] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12201.811760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12203.819408] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12213.927134] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12218.320941] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing lnet_if_list [12224.767609] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [12224.836727] LNet: Added LNI 192.168.203.130@tcp10 [8/256/0/180] [12231.282224] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [12231.341564] LNet: Added LNI 192.168.203.130@tcp11 [8/256/0/180] [12236.175938] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [12236.239686] LNet: Added LNI 192.168.203.130@tcp12 [8/256/0/180] [12239.933695] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [12239.956285] LNet: Added LNI 192.168.203.130@tcp13 [8/256/0/180] [12245.320903] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [12245.350420] LNet: Added LNI 192.168.203.130@tcp14 [8/256/0/180] [12249.424194] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [12249.478495] LNet: Added LNI 192.168.203.130@tcp15 [8/256/0/180] [12255.276125] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [12255.324492] LNet: Added LNI 192.168.203.130@tcp16 [8/256/0/180] [12260.273071] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [12260.310148] LNet: Added LNI 192.168.203.130@tcp17 [8/256/0/180] [12265.175834] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [12265.244573] LNet: Added LNI 192.168.203.130@tcp18 [8/256/0/180] [12270.075554] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [12270.112743] LNet: Added LNI 192.168.203.130@tcp19 [8/256/0/180] [12275.142439] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [12275.173230] LNet: Added LNI 192.168.203.130@tcp20 [8/256/0/180] [12280.979301] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [12280.998867] LNet: Added LNI 192.168.203.130@tcp21 [8/256/0/180] [12286.434196] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [12286.473152] LNet: Added LNI 192.168.203.130@tcp22 [8/256/0/180] [12292.077163] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [12292.121164] LNet: Added LNI 192.168.203.130@tcp23 [8/256/0/180] [12297.685192] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [12297.705668] LNet: Added LNI 192.168.203.130@tcp24 [8/256/0/180] [12303.349059] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [12303.392229] LNet: Added LNI 192.168.203.130@tcp25 [8/256/0/180] [12308.563910] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [12308.591563] LNet: Added LNI 192.168.203.130@tcp26 [8/256/0/180] [12315.079338] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [12315.122468] LNet: Added LNI 192.168.203.130@tcp27 [8/256/0/180] [12321.634734] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [12321.669576] LNet: Added LNI 192.168.203.130@tcp28 [8/256/0/180] [12327.702786] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [12327.737036] LNet: Added LNI 192.168.203.130@tcp29 [8/256/0/180] [12333.527435] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [12333.558354] LNet: Added LNI 192.168.203.130@tcp30 [8/256/0/180] [12339.425937] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [12339.455898] LNet: Added LNI 192.168.203.130@tcp31 [8/256/0/180] [12345.571414] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [12345.605342] LNet: Added LNI 192.168.203.130@tcp32 [8/256/0/180] [12351.432487] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [12351.452842] LNet: Added LNI 192.168.203.130@tcp33 [8/256/0/180] [12357.338968] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [12357.369076] LNet: Added LNI 192.168.203.130@tcp34 [8/256/0/180] [12364.119828] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [12364.152224] LNet: Added LNI 192.168.203.130@tcp35 [8/256/0/180] [12370.731176] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [12370.765506] LNet: Added LNI 192.168.203.130@tcp36 [8/256/0/180] [12377.523736] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [12377.551320] LNet: Added LNI 192.168.203.130@tcp37 [8/256/0/180] [12384.461210] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [12384.486851] LNet: Added LNI 192.168.203.130@tcp38 [8/256/0/180] [12390.481152] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [12390.511595] LNet: Added LNI 192.168.203.130@tcp39 [8/256/0/180] [12397.331409] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [12397.358547] LNet: Added LNI 192.168.203.130@tcp40 [8/256/0/180] [12406.385490] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [12406.437933] LNet: Added LNI 192.168.203.130@tcp41 [8/256/0/180] [12417.734449] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [12417.850250] LNet: Added LNI 192.168.203.130@tcp42 [8/256/0/180] [12427.410220] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [12427.465276] LNet: Added LNI 192.168.203.130@tcp43 [8/256/0/180] [12437.029815] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [12437.090491] LNet: Added LNI 192.168.203.130@tcp44 [8/256/0/180] [12446.711703] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [12446.761826] LNet: Added LNI 192.168.203.130@tcp45 [8/256/0/180] [12456.251546] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [12456.291667] LNet: Added LNI 192.168.203.130@tcp46 [8/256/0/180] [12464.396956] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [12464.445304] LNet: Added LNI 192.168.203.130@tcp47 [8/256/0/180] [12473.083304] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [12473.110691] LNet: Added LNI 192.168.203.130@tcp48 [8/256/0/180] [12482.565863] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [12482.616947] LNet: Added LNI 192.168.203.130@tcp49 [8/256/0/180] [12493.298939] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [12493.345895] LNet: Added LNI 192.168.203.130@tcp50 [8/256/0/180] [12514.280022] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12514.287054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12514.288867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12514.298079] Lustre: Skipped 2 previous similar messages [12514.310333] Lustre: Skipped 2 previous similar messages [12519.377647] Lustre: server umount lustre-MDT0000 complete [12523.665061] LustreError: 473237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788281726 with bad export cookie 7453913417744660362 [12523.675176] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12523.677678] LustreError: 473237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12524.271580] Lustre: server umount lustre-MDT0001 complete [12540.641441] Lustre: 472521:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788281727/real 1788281727] req@000000007ebda724 x1875148891397120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788281743 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12540.655984] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12543.392309] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [12544.993284] Lustre: 472518:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788281731/real 1788281731] req@00000000b7852543 x1875148891397248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788281747 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12546.081386] Lustre: 472518:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788281732/real 1788281732] req@00000000934b3180 x1875148891397376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788281748 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12550.112394] Lustre: 472520:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788281736/real 1788281736] req@0000000063f091af x1875148891397632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788281752 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12552.299534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12555.297145] Lustre: 472520:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788281741/real 1788281741] req@000000001fb8a65c x1875148891397760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788281757 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12569.318901] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x67719ec2cb3041e0 [12569.336218] Lustre: MGC192.168.203.130@tcp: Connection restored to 0@lo (at 0@lo) [12569.606805] LustreError: 475214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12569.640963] LustreError: 475214:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12569.675918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12570.721895] LustreError: 475215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12570.724522] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12571.747181] LustreError: 475214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12574.298727] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12577.251211] LustreError: 475215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12582.392687] LustreError: 475214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12583.154287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12583.443823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12585.528405] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12588.198320] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12595.918572] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12601.852444] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12621.286539] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12621.292814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12621.302633] Lustre: Skipped 1 previous similar message [12621.315537] Lustre: Skipped 1 previous similar message [12626.414646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12626.425385] Lustre: Skipped 1 previous similar message [12627.475704] Lustre: server umount lustre-OST0000 complete [12631.524736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12631.527128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12631.537081] Lustre: Skipped 1 previous similar message [12631.554421] Lustre: Skipped 1 previous similar message [12636.642402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12636.647301] Lustre: Skipped 1 previous similar message [12637.802750] Lustre: server umount lustre-MDT0000 complete [12641.763190] LustreError: 499459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12641.779570] LustreError: 499459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12641.980299] LustreError: 473238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788281844 with bad export cookie 7453913417744663008 [12641.994345] LustreError: 473238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12641.996217] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12642.485878] Lustre: server umount lustre-MDT0001 complete [12655.257746] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [12658.188293] Key type lgssc unregistered [12658.551530] LNet: 502245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12658.565132] LNetError: 502245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12658.572987] LNet: Removed LNI 192.168.203.130@tcp [12658.575652] LNet: Removed LNI 192.168.203.130@tcp10 [12658.581886] LNet: Removed LNI 192.168.203.130@tcp11 [12658.584905] LNet: Removed LNI 192.168.203.130@tcp12 [12658.592576] LNet: Removed LNI 192.168.203.130@tcp13 [12658.595776] LNet: Removed LNI 192.168.203.130@tcp14 [12658.598822] LNet: Removed LNI 192.168.203.130@tcp15 [12658.602436] LNet: Removed LNI 192.168.203.130@tcp16 [12658.605566] LNet: Removed LNI 192.168.203.130@tcp17 [12658.608585] LNet: Removed LNI 192.168.203.130@tcp18 [12658.611937] LNet: Removed LNI 192.168.203.130@tcp19 [12658.614980] LNet: Removed LNI 192.168.203.130@tcp20 [12658.617971] LNet: Removed LNI 192.168.203.130@tcp21 [12658.623555] LNet: Removed LNI 192.168.203.130@tcp22 [12658.626741] LNet: Removed LNI 192.168.203.130@tcp23 [12658.633320] LNet: Removed LNI 192.168.203.130@tcp24 [12658.645601] LNet: Removed LNI 192.168.203.130@tcp25 [12658.649507] LNet: Removed LNI 192.168.203.130@tcp26 [12658.653346] LNet: Removed LNI 192.168.203.130@tcp27 [12658.657479] LNet: Removed LNI 192.168.203.130@tcp28 [12658.659750] LNet: Removed LNI 192.168.203.130@tcp29 [12658.663743] LNet: Removed LNI 192.168.203.130@tcp30 [12658.665989] LNet: Removed LNI 192.168.203.130@tcp31 [12658.669927] LNet: Removed LNI 192.168.203.130@tcp32 [12658.673405] LNet: Removed LNI 192.168.203.130@tcp33 [12658.676676] LNet: Removed LNI 192.168.203.130@tcp34 [12658.679585] LNet: Removed LNI 192.168.203.130@tcp35 [12658.682047] LNet: Removed LNI 192.168.203.130@tcp36 [12658.687605] LNet: Removed LNI 192.168.203.130@tcp37 [12658.691321] LNet: Removed LNI 192.168.203.130@tcp38 [12658.694471] LNet: Removed LNI 192.168.203.130@tcp39 [12658.707239] LNet: Removed LNI 192.168.203.130@tcp40 [12658.710575] LNet: Removed LNI 192.168.203.130@tcp41 [12658.715637] LNet: Removed LNI 192.168.203.130@tcp42 [12658.722239] LNet: Removed LNI 192.168.203.130@tcp43 [12658.726027] LNet: Removed LNI 192.168.203.130@tcp44 [12658.729827] LNet: Removed LNI 192.168.203.130@tcp45 [12658.732949] LNet: Removed LNI 192.168.203.130@tcp46 [12658.737610] LNet: Removed LNI 192.168.203.130@tcp47 [12658.741683] LNet: Removed LNI 192.168.203.130@tcp48 [12658.744908] LNet: Removed LNI 192.168.203.130@tcp49 [12658.757460] LNet: Removed LNI 192.168.203.130@tcp50 [12659.581550] Key type .llcrypt unregistered [12659.583512] Key type ._llcrypt unregistered [12680.997596] Key type ._llcrypt registered [12681.002878] Key type .llcrypt registered [12681.147845] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [12697.254722] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [12698.209501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12698.271065] alg: No test for adler32 (adler32-zlib) [12699.353557] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [12699.625425] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [12701.328264] Key type lgssc registered [12702.202159] Lustre: Echo OBD driver; http://www.lustre.org/ [12738.276597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12738.309566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12739.746251] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12739.819754] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12740.006776] Lustre: lustre-MDT0000: new disk, initializing [12740.170376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12740.201646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12744.970901] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12754.969386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12755.095160] Lustre: 505443:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12755.137161] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12755.145139] Lustre: Skipped 1 previous similar message [12755.209632] Lustre: lustre-MDT0001: new disk, initializing [12755.272701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12755.298050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12755.312241] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12759.724284] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12769.529471] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12775.844958] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12783.112694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12783.343984] Lustre: *** cfs_fail_loc=90f, val=0*** [12783.425378] Lustre: lustre-OST0000: new disk, initializing [12783.428100] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12783.444688] Lustre: 507458:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12783.545860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12791.170520] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12792.435306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12792.448308] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12792.640819] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12803.333127] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12813.286612] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12813.288691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12813.295305] LustreError: Skipped 1 previous similar message [12813.297419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12813.320444] Lustre: Skipped 1 previous similar message [12818.404203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12818.427777] Lustre: Skipped 2 previous similar messages [12823.540295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12823.556988] Lustre: Skipped 1 previous similar message [12825.056120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12825.351779] Lustre: server umount lustre-OST0000 complete [12832.230441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12832.263725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12832.296321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12833.762444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12838.883682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12838.901180] Lustre: Skipped 3 previous similar messages [12845.024219] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12845.234980] Lustre: server umount lustre-MDT0000 complete [12849.212507] LustreError: 505434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788282051 with bad export cookie 16950839502606941197 [12849.232764] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12849.844921] Lustre: server umount lustre-MDT0001 complete [12864.021607] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [12867.053431] Key type lgssc unregistered [12867.417744] LNet: 509383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12867.427232] LNetError: 509383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12867.448797] LNet: Removed LNI 192.168.203.130@tcp [12868.309719] Key type .llcrypt unregistered [12868.312944] Key type ._llcrypt unregistered [12887.081244] Key type ._llcrypt registered [12887.085662] Key type .llcrypt registered [12887.236654] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [12888.400184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12888.420237] alg: No test for adler32 (adler32-zlib) [12889.749761] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [12890.111538] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [12891.816267] Key type lgssc registered [12893.197482] Lustre: Echo OBD driver; http://www.lustre.org/ [12902.914401] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12902.929358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12904.447248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12910.611049] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12914.669306] LustreError: 510653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12914.706451] LustreError: 510653:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12919.796593] LustreError: 510654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12922.938730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12923.476846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12929.212954] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12938.869882] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12946.613611] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12954.924602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12955.554750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12963.514339] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12973.504761] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12987.695574] loop0: detected capacity change from 0 to 204800000 [12987.898979] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12987.913416] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12987.928596] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12994.175355] loop0: detected capacity change from 0 to 204800000 [12994.414562] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12994.426022] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12994.448272] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12999.799085] loop0: detected capacity change from 0 to 204800000 [13004.629805] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13004.786751] Lustre: fs15246-OST0000: new disk, initializing [13004.790415] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [13004.795398] Lustre: 514246:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13004.835449] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13010.822337] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [13019.429752] loop1: detected capacity change from 0 to 204800000 [13025.390706] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13025.504835] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [13025.551354] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [13025.627286] Lustre: fs15246-MDT0000: new disk, initializing [13025.787690] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13025.822113] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13025.972703] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [13027.042967] Lustre: 515370:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000005db2aacc x1875149682950656/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788282245 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [13030.807729] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [13032.106368] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13032.129256] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13032.177661] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13056.488015] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13056.519253] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [13058.255572] Lustre: server umount fs15246-MDT0000 complete [13061.652422] LustreError: 510648:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13066.728219] LustreError: 510648:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13070.709738] Lustre: server umount fs15246-OST0000 complete [13085.665668] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13085.690233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13085.712065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13087.205859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13087.214727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13090.597184] Lustre: server umount lustre-OST0000 complete [13097.443127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13097.444234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13097.461560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13097.461569] Lustre: Skipped 1 previous similar message [13097.467816] Lustre: Skipped 1 previous similar message [13100.866682] Lustre: server umount lustre-MDT0000 complete [13102.575538] LustreError: 510654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13102.608700] LustreError: 510654:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13104.484052] LustreError: 510634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788282307 with bad export cookie 6248925813549646333 [13104.493249] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13104.505564] LustreError: 510634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13104.795829] Lustre: server umount lustre-MDT0001 complete [13115.515268] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [13118.872943] Key type lgssc unregistered [13119.253079] LNet: 518471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13119.270834] LNetError: 518471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13119.296113] LNet: Removed LNI 192.168.203.130@tcp [13120.224619] Key type .llcrypt unregistered [13120.232746] Key type ._llcrypt unregistered [13138.911576] Key type ._llcrypt registered [13138.917754] Key type .llcrypt registered [13139.066898] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [13139.984709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13140.246127] alg: No test for adler32 (adler32-zlib) [13141.379365] Lustre: Lustre: Build Version: 2.17.57_82_g90dbd01 [13141.658607] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [13143.328171] Key type lgssc registered [13144.603769] Lustre: Echo OBD driver; http://www.lustre.org/ [13185.960974] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [13197.454236] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [13233.674981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13233.711498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13234.904412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13234.951051] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13235.038962] Lustre: lustre-MDT0000: new disk, initializing [13235.129620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13235.162551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13239.080382] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [13249.050369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13249.192377] Lustre: 522941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13249.243871] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13249.249263] Lustre: Skipped 1 previous similar message [13249.365786] Lustre: lustre-MDT0001: new disk, initializing [13249.470830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13249.500668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13249.517995] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13254.080786] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [13263.989400] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13270.544731] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13279.492503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13280.037583] Lustre: lustre-OST0000: new disk, initializing [13280.046537] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13280.058164] Lustre: 524911:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13280.177286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13287.261183] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [13289.020286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13289.035255] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13289.158497] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13297.445508] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13302.288791] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13302.497619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13307.544807] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13307.792684] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13309.938868] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13309.947923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13309.948074] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13309.967044] Lustre: Skipped 1 previous similar message [13309.993335] Lustre: Skipped 1 previous similar message [13315.043381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13315.049264] Lustre: Skipped 1 previous similar message [13315.858459] Lustre: server umount lustre-OST0000 complete [13320.164350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13320.188396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13320.198028] Lustre: Skipped 1 previous similar message [13325.282836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13325.290953] Lustre: Skipped 1 previous similar message [13325.919387] Lustre: server umount lustre-MDT0000 complete [13329.535354] LustreError: 522934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788282532 with bad export cookie 16809780497143893199 [13329.539032] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13329.543601] LustreError: 522934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13329.928035] Lustre: server umount lustre-MDT0001 complete [13355.671722] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [13369.842289] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [13412.381545] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13255 sec ========= 13:10:13 (1788282613) [13414.982912] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:10:15 (1788282615) === [13419.324781] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:10:20 (1788282620) === [13466.852649] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [13471.221223] Key type lgssc unregistered [13471.709898] LNet: 531563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13471.722071] LNetError: 531563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13471.753711] LNet: Removed LNI 192.168.203.130@tcp [13473.146592] Key type .llcrypt unregistered [13473.151623] Key type ._llcrypt unregistered