[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 452755880 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003292] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009005] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010026] pid_max: default: 32768 minimum: 301 [ 0.011162] LSM: Security Framework initializing [ 0.012057] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014105] *** VALIDATE selinux *** [ 0.021729] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025805] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026186] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028134] *** VALIDATE tmpfs *** [ 0.030239] *** VALIDATE proc *** [ 0.031215] *** VALIDATE cgroup *** [ 0.032011] *** VALIDATE cgroup2 *** [ 0.033283] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034183] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036044] Spectre V2 : User space: Vulnerable [ 0.037014] Speculative Store Bypass: Vulnerable [ 0.040591] debug: unmapping init [mem 0xffffffff9f859000-0xffffffff9f860fff] [ 0.042208] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043761] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044032] ... version: 2 [ 0.045017] ... bit width: 48 [ 0.046013] ... generic registers: 4 [ 0.047013] ... value mask: 0000ffffffffffff [ 0.048016] ... max period: 00007fffffffffff [ 0.049017] ... fixed-purpose events: 3 [ 0.050012] ... event mask: 000000070000000f [ 0.052272] rcu: Hierarchical SRCU implementation. [ 0.054642] smp: Bringing up secondary CPUs ... [ 0.055670] x86: Booting SMP configuration: [ 0.056036] .... node #0, CPUs: #1 #2 #3 [ 0.059623] smp: Brought up 1 node, 4 CPUs [ 0.061014] smpboot: Max logical packages: 1 [ 0.062017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.247031] node 0 deferred pages initialised in 182ms [ 0.250190] devtmpfs: initialized [ 0.251236] x86/mm: Memory block size: 128MB [ 0.253310] gcov: version magic: 0x41383552 [ 0.257022] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.261092] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.264488] pinctrl core: initialized pinctrl subsystem [ 0.266261] [ 0.266777] ************************************************************* [ 0.269011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.271013] ** ** [ 0.274015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.276021] ** ** [ 0.278012] ** This means that this kernel is built to expose internal ** [ 0.280012] ** IOMMU data structures, which may compromise security on ** [ 0.282014] ** your system. ** [ 0.285016] ** ** [ 0.287015] ** If you see this message and you are not debugging the ** [ 0.290016] ** kernel, report this immediately to your vendor! ** [ 0.292014] ** ** [ 0.294017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.296014] ************************************************************* [ 0.298757] NET: Registered protocol family 16 [ 0.300539] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.304069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.306048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.309040] cpuidle: using governor menu [ 0.311000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.314602] PCI: Using configuration type 1 for base access [ 0.317216] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.327114] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.330093] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.334133] cryptd: max_cpu_qlen set to 1000 [ 0.342493] ACPI: Added _OSI(Module Device) [ 0.344016] ACPI: Added _OSI(Processor Device) [ 0.345011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.346009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.349381] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.356554] ACPI: Interpreter enabled [ 0.357057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.358014] ACPI: Using IOAPIC for interrupt routing [ 0.360083] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.365574] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.376625] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.380051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.382016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.384073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.388559] acpiphp: Slot [2] registered [ 0.390130] acpiphp: Slot [5] registered [ 0.392191] acpiphp: Slot [6] registered [ 0.393118] acpiphp: Slot [7] registered [ 0.394134] acpiphp: Slot [8] registered [ 0.397141] acpiphp: Slot [9] registered [ 0.398155] acpiphp: Slot [10] registered [ 0.400136] acpiphp: Slot [3] registered [ 0.401101] acpiphp: Slot [4] registered [ 0.403107] acpiphp: Slot [11] registered [ 0.404094] acpiphp: Slot [12] registered [ 0.406111] acpiphp: Slot [13] registered [ 0.407141] acpiphp: Slot [14] registered [ 0.409204] acpiphp: Slot [15] registered [ 0.411135] acpiphp: Slot [16] registered [ 0.413123] acpiphp: Slot [17] registered [ 0.415116] acpiphp: Slot [18] registered [ 0.417100] acpiphp: Slot [19] registered [ 0.418112] acpiphp: Slot [20] registered [ 0.419086] acpiphp: Slot [21] registered [ 0.421171] acpiphp: Slot [22] registered [ 0.422107] acpiphp: Slot [23] registered [ 0.424099] acpiphp: Slot [24] registered [ 0.426095] acpiphp: Slot [25] registered [ 0.427117] acpiphp: Slot [26] registered [ 0.428088] acpiphp: Slot [27] registered [ 0.430109] acpiphp: Slot [28] registered [ 0.431099] acpiphp: Slot [29] registered [ 0.433101] acpiphp: Slot [30] registered [ 0.434121] acpiphp: Slot [31] registered [ 0.436048] PCI host bridge to bus 0000:00 [ 0.437013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.440030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.442027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.445029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.447028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.451037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.453193] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.457098] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.461570] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.470016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.475177] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.477019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.479018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.482018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.484776] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.487729] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.490043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.492816] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.496749] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.508016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.515014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.520000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.531019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.538018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.556024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.570316] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.579015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.584016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.603014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.612785] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.618015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.624019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.643020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.653088] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.662019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.668017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.689017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.701163] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.707019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.714018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.729017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.739294] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.747020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.758018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.780015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.791156] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.793392] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.795347] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.797345] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.799650] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.805133] iommu: Default domain type: Passthrough [ 0.806489] SCSI subsystem initialized [ 0.807129] ACPI: bus type USB registered [ 0.808165] usbcore: registered new interface driver usbfs [ 0.809099] usbcore: registered new interface driver hub [ 0.810111] usbcore: registered new device driver usb [ 0.811179] pps_core: LinuxPPS API ver. 1 registered [ 0.813015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.816125] PTP clock support registered [ 0.818303] EDAC MC: Ver: 3.0.0 [ 0.821154] PCI: Using ACPI for IRQ routing [ 0.824660] NetLabel: Initializing [ 0.825013] NetLabel: domain hash size = 128 [ 0.827010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.828243] NetLabel: unlabeled traffic allowed by default [ 0.831041] vgaarb: loaded [ 0.832282] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.833185] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.843000] clocksource: Switched to clocksource kvm-clock [ 0.952578] VFS: Disk quotas dquot_6.6.0 [ 0.954111] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.957170] *** VALIDATE ramfs *** [ 0.958599] *** VALIDATE hugetlbfs *** [ 0.961398] pnp: PnP ACPI init [ 0.964330] pnp: PnP ACPI: found 6 devices [ 0.981138] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.984247] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.986668] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.988843] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.991594] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.994279] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.997693] NET: Registered protocol family 2 [ 1.000272] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.004717] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.007862] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.012398] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.015291] TCP: Hash tables configured (established 65536 bind 65536) [ 1.017659] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.020137] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.039889] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.046521] NET: Registered protocol family 1 [ 1.051984] RPC: Registered named UNIX socket transport module. [ 1.054530] RPC: Registered udp transport module. [ 1.056685] RPC: Registered tcp transport module. [ 1.058658] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.061654] NET: Registered protocol family 44 [ 1.064447] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.067293] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.069459] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.071546] PCI: CLS 0 bytes, default 64 [ 1.073263] Unpacking initramfs... [ 2.454140] debug: unmapping init [mem 0xffff88dd7cc54000-0xffff88dd7ffbffff] [ 2.458589] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.461099] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.464628] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.980330] Initialise system trusted keyrings [ 2.982246] Key type blacklist registered [ 2.984181] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.992976] zbud: loaded [ 2.996376] *** VALIDATE nfs *** [ 2.997668] *** VALIDATE nfs4 *** [ 2.999324] pstore: using deflate compression [ 3.003163] Platform Keyring initialized [ 3.118920] NET: Registered protocol family 38 [ 3.122654] Key type asymmetric registered [ 3.124116] Asymmetric key parser 'x509' registered [ 3.126122] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.129079] io scheduler mq-deadline registered [ 3.130863] io scheduler kyber registered [ 3.132518] io scheduler bfq registered [ 3.134760] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.137863] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.143289] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.147040] ACPI: Power Button [PWRF] [ 3.153973] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.161271] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.178629] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.185089] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.201519] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.229498] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.257411] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.261674] Non-volatile memory driver v1.3 [ 3.263485] Linux agpgart interface v0.103 [ 3.291265] virtio_blk virtio1: [vda] 146648 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.293986] vda: detected capacity change from 0 to 75083776 [ 3.308609] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.310638] vdb: detected capacity change from 0 to 1073741824 [ 3.327944] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.334042] vdc: detected capacity change from 0 to 2621440000 [ 3.351802] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.354489] vdd: detected capacity change from 0 to 2621440000 [ 3.377154] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.381019] vde: detected capacity change from 0 to 4294967296 [ 3.399200] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.402498] vdf: detected capacity change from 0 to 4294967296 [ 3.410914] libphy: Fixed MDIO Bus: probed [ 3.417825] usbcore: registered new interface driver usbserial_generic [ 3.420074] usbserial: USB Serial support registered for generic [ 3.422349] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.426191] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.428128] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.431776] mousedev: PS/2 mouse device common for all mice [ 3.435746] rtc_cmos 00:05: RTC can wake from S4 [ 3.438705] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.439074] rtc_cmos 00:05: registered as rtc0 [ 3.444758] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.447881] intel_pstate: CPU model not supported [ 3.449754] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.457546] hid: raw HID events driver (C) Jiri Kosina [ 3.462836] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.463268] usbcore: registered new interface driver usbhid [ 3.468516] usbhid: USB HID core driver [ 3.469992] drop_monitor: Initializing network drop monitor service [ 3.472158] Initializing XFRM netlink socket [ 3.473947] NET: Registered protocol family 10 [ 3.476515] Segment Routing with IPv6 [ 3.477883] NET: Registered protocol family 17 [ 3.479632] mpls_gso: MPLS GSO support [ 3.486579] RAS: Correctable Errors collector initialized. [ 3.490695] AVX version of gcm_enc/dec engaged. [ 3.492514] AES CTR mode by8 optimization enabled [ 3.584709] sched_clock: Marking stable (3584681922, 0)->(4494226733, -909544811) [ 3.588476] registered taskstats version 1 [ 3.590898] Loading compiled-in X.509 certificates [ 3.593244] zswap: loaded using pool lzo/zbud [ 3.625654] Key type big_key registered [ 3.645394] Key type encrypted registered [ 3.646986] ima: No TPM chip found, activating TPM-bypass! [ 3.649677] ima: Allocated hash algorithm: sha1 [ 3.651612] ima: No architecture policies found [ 3.653702] evm: Initialising EVM extended attributes: [ 3.655194] evm: security.selinux [ 3.656479] evm: security.ima [ 3.657493] evm: security.capability [ 3.658863] evm: HMAC attrs: 0x1 [ 3.661342] rtc_cmos 00:05: setting system clock to 2026-09-08 11:13:54 UTC (1788866034) [ 3.667180] debug: unmapping init [mem 0xffffffffa0803000-0xffffffffa09fffff] [ 3.671121] debug: unmapping init [mem 0xffffffff9f582000-0xffffffff9f858fff] [ 3.684068] Write protecting the kernel read-only data: 28672k [ 3.687663] debug: unmapping init [mem 0xffffffff9dc03000-0xffffffff9ddfffff] [ 3.690753] debug: unmapping init [mem 0xffffffff9e514000-0xffffffff9e5fffff] [ 3.726265] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.735184] systemd[1]: Detected virtualization kvm. [ 3.736905] systemd[1]: Detected architecture x86-64. [ 3.738809] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.764893] systemd[1]: No hostname configured. [ 3.767265] systemd[1]: Set hostname to . [ 3.769518] random: systemd: uninitialized urandom read (16 bytes read) [ 3.772411] systemd[1]: Initializing machine ID from random generator. [ 3.830404] random: ln: uninitialized urandom read (6 bytes read) [ 3.913033] random: systemd: uninitialized urandom read (16 bytes read) [ 3.916233] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.921278] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.925700] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.580296] device-mapper: uevent: version 1.0.3 [ 4.582707] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.310484] virtio_net virtio0 ens2: renamed from eth0 [ 5.342229] random: fast init done [ 5.398443] scsi host0: ata_piix [ 5.419971] scsi host1: ata_piix [ 5.422694] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.425024] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.232617] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.165534] random: crng init done [ 10.167880] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.684203] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.929463] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.292804] SELinux: Disabled at runtime. [ 12.363201] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.380370] systemd[1]: Detected virtualization kvm. [ 12.382445] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.985674] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.994090] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.005804] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.013466] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.017667] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.028230] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.032642] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ 13.224426] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.562272] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.904718] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.983763] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.084984] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.094645] EDAC sbridge: Ver: 1.1.2 [ 15.901266] Key type dns_resolver registered [ 16.204412] NFS: Registering the id_resolver key type [ 16.206623] Key type id_resolver registered [ 16.208012] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg652-server login: [ 42.528542] libcfs: loading out-of-tree module taints kernel. [ 42.547272] Key type ._llcrypt registered [ 42.548573] Key type .llcrypt registered [ 42.591492] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 50.621888] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 51.224848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 51.233080] alg: No test for adler32 (adler32-zlib) [ 52.216267] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 52.500189] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 54.119164] Key type lgssc registered [ 54.760586] Lustre: Echo OBD driver; http://www.lustre.org/ [ 61.574461] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 76.324297] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 81.077419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 81.088067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 82.182654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 82.195774] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 82.238573] Lustre: lustre-MDT0000: new disk, initializing [ 82.267539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 82.277308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 83.815184] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 90.026194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 90.077806] Lustre: 6475:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 90.096850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 90.100098] Lustre: Skipped 1 previous similar message [ 90.147699] Lustre: lustre-MDT0001: new disk, initializing [ 90.188503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 90.210263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 90.216389] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 92.085928] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 94.704933] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 98.675965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 98.784919] Lustre: lustre-OST0000: new disk, initializing [ 98.787572] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 98.791750] Lustre: 8411:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 98.814784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 99.175233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 99.180463] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 99.203953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 101.045301] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 107.395195] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 107.446174] Lustre: lustre-OST0001: new disk, initializing [ 107.448384] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 107.451659] Lustre: 9483:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 107.475876] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 109.535353] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 113.131120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 113.134199] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 113.149500] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 115.555922] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 119.209936] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 121.146267] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing check_logdir /tmp/testlogs/ [ 122.595953] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing yml_node [ 124.180127] Lustre: DEBUG MARKER: Client: 2.17.57.104 [ 125.077627] Lustre: DEBUG MARKER: MDS: 2.17.57.104 [ 125.990662] Lustre: DEBUG MARKER: OSS: 2.17.57.104 [ 126.581689] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Sep 8 07:15:55 EDT 2026 [ 132.982940] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 133.614215] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 146.399780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 146.402857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 146.408157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 148.960058] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 148.960429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 148.967398] Lustre: Skipped 2 previous similar messages [ 148.969777] Lustre: Skipped 2 previous similar messages [ 151.713300] Lustre: server umount lustre-MDT0000 complete [ 153.109960] LustreError: 10299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866183 with bad export cookie 12208450665670940110 [ 153.112284] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 153.114751] LustreError: 10299:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 154.082692] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 154.083431] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 154.091443] Lustre: Skipped 1 previous similar message [ 158.112476] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 158.116190] Lustre: Skipped 2 previous similar messages [ 162.207955] LustreError: 6483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 162.207985] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 162.215520] LustreError: 6483:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 162.218040] Lustre: Skipped 1 previous similar message [ 167.391830] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 167.392434] LustreError: 6481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 167.408785] LustreError: 6481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 167.488625] Lustre: server umount lustre-MDT0001 complete [ 175.306312] Lustre: server umount lustre-OST0000 complete [ 183.054550] Lustre: server umount lustre-OST0001 complete [ 185.318205] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 188.649504] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 201.745107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 201.836405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 201.850521] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 201.890070] Lustre: lustre-MDT0000: new disk, initializing [ 201.921543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 201.930951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 203.502826] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 208.244208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 208.299811] Lustre: 15024:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 208.426984] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 209.999372] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 214.168310] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 216.253902] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 218.671400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 218.775429] Lustre: lustre-OST0000: new disk, initializing [ 218.776865] Lustre: Skipped 1 previous similar message [ 218.778487] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 218.780657] Lustre: Skipped 2 previous similar messages [ 218.782676] Lustre: 16993:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 218.813168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 218.816650] Lustre: Skipped 1 previous similar message [ 220.359368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 220.364015] Lustre: Skipped 1 previous similar message [ 220.366714] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 220.403152] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 220.929520] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 224.803453] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 226.279545] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 226.356866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 227.809147] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 227.888960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 230.880396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 230.882326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 230.883150] LustreError: Skipped 1 previous similar message [ 230.883791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 230.887013] Lustre: Skipped 1 previous similar message [ 230.888774] Lustre: Skipped 2 previous similar messages [ 234.709744] Lustre: server umount lustre-OST0000 complete [ 239.072130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 239.075585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 250.335106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 250.336665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 250.340867] Lustre: Skipped 6 previous similar messages [ 250.393087] Lustre: server umount lustre-MDT0000 complete [ 251.668269] LustreError: 15823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866282 with bad export cookie 12208450665670941629 [ 251.671384] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 251.673774] LustreError: 15823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 251.789836] Lustre: server umount lustre-MDT0001 complete [ 253.239126] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 07:18:02 (1788866282) [ 254.110259] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 254.760618] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 07:18:04 (1788866284) [ 257.261614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 257.441311] LustreError: 19399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 257.472515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 258.959566] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 262.169933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 263.848731] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 266.791663] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 269.189001] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 272.212534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 274.810758] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 278.227543] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 287.716989] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 287.717641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 287.718072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 287.722022] Lustre: Skipped 2 previous similar messages [ 292.375669] Lustre: server umount lustre-OST0000 complete [ 300.127656] Lustre: server umount lustre-MDT0000 complete [ 301.690536] LustreError: 19424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866332 with bad export cookie 12208450665670942511 [ 301.693825] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 301.698153] LustreError: 19424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 301.866307] Lustre: server umount lustre-MDT0001 complete [ 306.255282] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 307.548727] Key type lgssc unregistered [ 307.700198] LNet: 22927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 307.707094] LNetError: 22927:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 307.715250] LNet: Removed LNI 192.168.206.152@tcp [ 308.166459] Key type .llcrypt unregistered [ 308.168705] Key type ._llcrypt unregistered [ 318.376423] Key type ._llcrypt registered [ 318.377864] Key type .llcrypt registered [ 318.431264] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 318.793613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 318.815799] alg: No test for adler32 (adler32-zlib) [ 319.721556] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 319.836504] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 321.431291] Key type lgssc registered [ 321.977578] Lustre: Echo OBD driver; http://www.lustre.org/ [ 325.667205] loop: module loaded [ 325.694362] loop0: detected capacity change from 0 to 204800000 [ 325.805238] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 325.812286] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 325.957289] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 328.260493] loop0: detected capacity change from 0 to 204800000 [ 328.320524] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 328.325167] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 328.329898] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 330.568827] loop0: detected capacity change from 0 to 204800000 [ 332.955955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 332.968831] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 334.073173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 334.091265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 334.133245] Lustre: lustre-MDT0000: new disk, initializing [ 334.169986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 334.179694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 335.942501] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 339.950906] loop1: detected capacity change from 0 to 204800000 [ 342.048950] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 342.172262] Lustre: lustre-OST07c6: new disk, initializing [ 342.174576] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 342.176933] Lustre: Skipped 1 previous similar message [ 342.179293] Lustre: 26089:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 342.215129] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 344.026173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 344.032547] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 344.054792] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 344.660229] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 346.939338] Lustre: 25070:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 346.943846] Lustre: 25070:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 346.948033] Lustre: 25070:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 346.951912] Lustre: 25070:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 346.955603] Lustre: 25070:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 346.958706] Lustre: 25070:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 364.562563] Lustre: 25072:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 364.578343] Lustre: 25072:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 364.584534] Lustre: 25072:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 364.596122] Lustre: 25072:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 364.604685] Lustre: 25072:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 364.612193] Lustre: 25072:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 368.642552] Lustre: 25100:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 368.648036] Lustre: 25100:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 368.653627] Lustre: 25100:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 368.657541] Lustre: 25100:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 368.665015] Lustre: 25100:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 368.669178] Lustre: 25100:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 368.676811] Lustre: 25100:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 368.682135] Lustre: 25100:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 368.687438] Lustre: 25100:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 368.691249] Lustre: 25100:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 368.695215] Lustre: 25100:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 368.699375] Lustre: 25100:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 371.848190] Lustre: 25072:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 371.856085] Lustre: 25072:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 371.862421] Lustre: 25072:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 371.868742] Lustre: 25072:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 371.872659] Lustre: 25072:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 371.875814] Lustre: 25072:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 371.879432] Lustre: 25072:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 371.882861] Lustre: 25072:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 371.886433] Lustre: 25072:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 371.890237] Lustre: 25072:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 371.893982] Lustre: 25072:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 371.897478] Lustre: 25072:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 378.874048] Lustre: 25071:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 378.879916] Lustre: 25071:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 378.886728] Lustre: 25071:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 378.890968] Lustre: 25071:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 378.894723] Lustre: 25071:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 378.898376] Lustre: 25071:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 378.902105] Lustre: 25071:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 378.906308] Lustre: 25071:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 378.910133] Lustre: 25071:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 378.913258] Lustre: 25071:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 378.916769] Lustre: 25071:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 378.920271] Lustre: 25071:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 388.736749] Lustre: 25070:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 388.741548] Lustre: 25070:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 388.745217] Lustre: 25070:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 388.747888] Lustre: 25070:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 388.751191] Lustre: 25070:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 388.754630] Lustre: 25070:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 388.757692] Lustre: 25070:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 388.762218] Lustre: 25070:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 388.766665] Lustre: 25070:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 388.770271] Lustre: 25070:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 388.773629] Lustre: 25070:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 388.777620] Lustre: 25070:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 431.071997] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 431.075385] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 431.083278] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 434.144905] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 439.265697] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 444.383121] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 444.386712] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 444.460361] Lustre: server umount lustre-OST07c6 complete [ 449.030316] Lustre: server umount lustre-MDT0000 complete [ 456.334780] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 457.676636] Key type lgssc unregistered [ 457.823264] LNet: 28646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 457.826720] LNetError: 28646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 457.840339] LNet: Removed LNI 192.168.206.152@tcp [ 458.191820] Key type .llcrypt unregistered [ 458.193394] Key type ._llcrypt unregistered [ 466.927696] Key type ._llcrypt registered [ 466.929318] Key type .llcrypt registered [ 466.977931] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 467.443789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 467.450973] alg: No test for adler32 (adler32-zlib) [ 468.343802] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 468.457186] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 470.071186] Key type lgssc registered [ 470.524979] Lustre: Echo OBD driver; http://www.lustre.org/ [ 474.075995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 474.096779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 475.332070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 477.117390] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 481.103952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 481.319252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 483.212709] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 487.335539] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 490.346790] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 493.550068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 493.713889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 496.347392] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 499.977826] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 500.902301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 517.600864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 517.601031] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 517.608125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 517.614834] Lustre: Skipped 1 previous similar message [ 522.722731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 522.725086] Lustre: Skipped 1 previous similar message [ 522.784134] Lustre: server umount lustre-OST0000 complete [ 527.331892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 527.335992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 527.343141] Lustre: Skipped 1 previous similar message [ 527.345598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 530.518238] Lustre: server umount lustre-MDT0000 complete [ 532.105458] LustreError: 29871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866562 with bad export cookie 17548975649340833758 [ 532.108859] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 532.113152] LustreError: 29871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 532.266463] Lustre: server umount lustre-MDT0001 complete [ 537.042386] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 538.433319] Key type lgssc unregistered [ 538.594359] LNet: 33640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 538.598621] LNetError: 33640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 538.609233] LNet: Removed LNI 192.168.206.152@tcp [ 539.055116] Key type .llcrypt unregistered [ 539.056636] Key type ._llcrypt unregistered [ 548.278086] Key type ._llcrypt registered [ 548.279254] Key type .llcrypt registered [ 548.325175] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 548.694220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 548.741030] alg: No test for adler32 (adler32-zlib) [ 549.641495] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 549.774283] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 551.367768] Key type lgssc registered [ 551.823341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 554.951461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 554.960227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 556.148114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 557.680051] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 561.004370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 561.161209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 562.728989] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 565.514465] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 567.827087] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 570.781465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 570.942073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 573.482254] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 576.678223] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 580.208655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 595.941859] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 595.942217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 595.949379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 595.950780] LustreError: Skipped 1 previous similar message [ 595.955299] Lustre: Skipped 1 previous similar message [ 597.137529] Lustre: server umount lustre-OST0000 complete [ 601.057053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 601.060648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 601.062173] Lustre: Skipped 2 previous similar messages [ 601.066023] Lustre: Skipped 1 previous similar message [ 606.177102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 606.179966] Lustre: Skipped 1 previous similar message [ 611.295945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 611.298156] Lustre: Skipped 1 previous similar message [ 613.343121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 613.408154] Lustre: server umount lustre-MDT0000 complete [ 614.771468] LustreError: 35626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866645 with bad export cookie 11262702246845580904 [ 614.773878] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 614.777388] LustreError: 35626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 614.911914] Lustre: server umount lustre-MDT0001 complete [ 619.065314] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 620.186937] Key type lgssc unregistered [ 620.335476] LNet: 38440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 620.339208] LNetError: 38440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 620.348426] LNet: Removed LNI 192.168.206.152@tcp [ 620.681463] Key type .llcrypt unregistered [ 620.683732] Key type ._llcrypt unregistered [ 630.659178] Key type ._llcrypt registered [ 630.660838] Key type .llcrypt registered [ 630.710244] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 631.139605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 631.158546] alg: No test for adler32 (adler32-zlib) [ 632.023434] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 632.120352] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 633.711230] Key type lgssc registered [ 634.081570] Lustre: Echo OBD driver; http://www.lustre.org/ [ 636.939435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 636.949230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 638.172375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 639.787062] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 643.087715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 643.225657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 644.914486] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 648.084505] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 650.761561] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 653.871869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 654.016611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 656.353831] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 659.510760] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 662.249884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 673.762934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 673.765165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 673.773718] Lustre: Skipped 1 previous similar message [ 676.251078] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.52@tcp (stopping) [ 676.254676] Lustre: Skipped 2 previous similar messages [ 678.522970] Lustre: server umount lustre-MDT0000 complete [ 678.883250] LustreError: 39832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 678.895731] LustreError: 39832:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 680.332581] LustreError: 39811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866711 with bad export cookie 13599082357915405451 [ 680.338819] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 681.354697] LustreError: 39828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 681.364342] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.52@tcp (stopping) [ 684.000388] LustreError: 40558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 684.000800] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 684.018752] Lustre: Skipped 1 previous similar message [ 684.020802] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 686.050747] LustreError: 41070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 686.538478] Lustre: server umount lustre-MDT0001 complete [ 697.681918] LustreError: 39812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.52@tcp arrived at 1788866728 with bad export cookie 13599082357915406095 [ 697.688646] LustreError: 39812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 713.183095] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 713.264661] Lustre: server umount lustre-OST0000 complete [ 720.346860] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 721.645314] Key type lgssc unregistered [ 721.780617] LNet: 43715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.784227] LNetError: 43715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 721.797703] LNet: Removed LNI 192.168.206.152@tcp [ 722.136260] Key type .llcrypt unregistered [ 722.137811] Key type ._llcrypt unregistered [ 732.604166] Key type ._llcrypt registered [ 732.605879] Key type .llcrypt registered [ 732.658145] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 733.145096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 733.159177] alg: No test for adler32 (adler32-zlib) [ 734.050849] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 734.160428] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 735.751746] Key type lgssc registered [ 736.241909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 740.177086] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 740.195433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 741.467115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 744.017455] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 748.650073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 748.854880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 751.190521] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 754.921980] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 757.642202] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 760.904401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 761.060650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 763.837744] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 766.186532] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 767.594545] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 776.671912] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 776.675305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 776.681322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 778.721534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 778.723696] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 778.728368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 781.204458] Lustre: server umount lustre-OST0000 complete [ 795.104038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 795.107088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 795.112065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 795.115202] Lustre: Skipped 2 previous similar messages [ 798.176300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 798.177646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 800.626884] Lustre: server umount lustre-MDT0000 complete [ 802.117992] LustreError: 45085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866832 with bad export cookie 1441495496489766446 [ 802.121345] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 802.124274] LustreError: 45085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 808.309142] Lustre: server umount lustre-MDT0001 complete [ 812.407870] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 813.562491] Key type lgssc unregistered [ 813.713390] LNet: 48807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.716699] LNetError: 48807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 813.728947] LNet: Removed LNI 192.168.206.152@tcp [ 814.103377] Key type .llcrypt unregistered [ 814.105110] Key type ._llcrypt unregistered [ 824.879743] Key type ._llcrypt registered [ 824.881496] Key type .llcrypt registered [ 824.929254] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 825.300535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 825.360146] alg: No test for adler32 (adler32-zlib) [ 826.258791] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 826.379927] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 827.983161] Key type lgssc registered [ 828.458610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 831.854769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 831.861846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 833.056312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 834.858402] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 838.469338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 838.644866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 840.312678] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 843.241940] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 845.380239] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 847.962468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 848.105958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 850.357653] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 853.296099] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 857.320196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 861.193138] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 862.646937] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 873.474240] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-08 7h28m22s [ 874.095473] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 878.048612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 878.050097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 878.055085] Lustre: Skipped 2 previous similar messages [ 878.058392] Lustre: Skipped 1 previous similar message [ 879.072985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 879.075533] Lustre: Skipped 2 previous similar messages [ 880.522883] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.52@tcp (stopping) [ 880.527577] Lustre: Skipped 1 previous similar message [ 885.217283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 885.220606] Lustre: Skipped 2 previous similar messages [ 889.311216] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 889.381737] Lustre: server umount lustre-MDT0000 complete [ 890.652591] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 891.360596] LustreError: 50928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 891.367423] LustreError: 50928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 893.143103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 893.204843] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 893.304210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 893.333098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 894.309173] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 894.885569] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 896.417205] Lustre: DEBUG MARKER: Wait for df (36552) ... [ 897.090760] Lustre: DEBUG MARKER: done [ 899.553312] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 899.562704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 899.567259] Lustre: Skipped 1 previous similar message [ 904.657145] Lustre: server umount lustre-OST0000 complete [ 909.792927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 909.794100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 909.794267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 909.794272] Lustre: Skipped 1 previous similar message [ 909.798251] Lustre: Skipped 2 previous similar messages [ 912.218140] Lustre: server umount lustre-MDT0000 complete [ 913.574514] LustreError: 50179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788866944 with bad export cookie 16835295779405560178 [ 913.576439] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 913.579862] LustreError: 50179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 913.704581] Lustre: server umount lustre-MDT0001 complete [ 917.703518] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 918.856359] Key type lgssc unregistered [ 918.994441] LNet: 55114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.998802] LNetError: 55114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.007769] LNet: Removed LNI 192.168.206.152@tcp [ 919.343491] Key type .llcrypt unregistered [ 919.345277] Key type ._llcrypt unregistered [ 934.687880] Key type ._llcrypt registered [ 934.690284] Key type .llcrypt registered [ 934.737740] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 935.219571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 935.232808] alg: No test for adler32 (adler32-zlib) [ 936.115569] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 936.214151] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 937.807199] Key type lgssc registered [ 938.219674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 941.227364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 941.233578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 942.332468] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 942.341948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 942.406462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 944.002058] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 947.634435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 947.716400] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 947.735811] Lustre: 57292:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 947.872768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 949.715073] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 952.720471] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 954.987753] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 957.600249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 957.680290] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 957.756975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 959.839266] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 959.966266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 962.630732] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 967.260465] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-08 7h29m56s [ 967.880276] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 970.675571] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 971.650102] Lustre: DEBUG MARKER: Creating a test file and stat it [ 971.668570] LustreError: 57300:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 992.159379] Lustre: lustre-MDT0000: Client 85dc2ecb-8636-423b-a6d4-69739e892b8d (at 192.168.206.52@tcp) reconnecting [ 1011.711418] LustreError: 57300:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1012.695148] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1013.807836] Lustre: DEBUG MARKER: done [ 1016.289744] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1016.290095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1016.304741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1016.309240] Lustre: Skipped 1 previous similar message [ 1021.412824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1021.415878] Lustre: Skipped 1 previous similar message [ 1021.979453] Lustre: server umount lustre-OST0000 complete [ 1024.992539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1024.995123] LustreError: Skipped 1 previous similar message [ 1024.996585] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1025.001951] Lustre: Skipped 1 previous similar message [ 1025.004579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1029.688502] Lustre: server umount lustre-MDT0000 complete [ 1031.261421] LustreError: 57287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867062 with bad export cookie 7240464326544011113 [ 1031.263708] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1031.267083] LustreError: 57287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1031.400856] Lustre: server umount lustre-MDT0001 complete [ 1035.467387] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 1036.575853] Key type lgssc unregistered [ 1036.713472] LNet: 61615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.716493] LNetError: 61615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1036.724742] LNet: Removed LNI 192.168.206.152@tcp [ 1037.045648] Key type .llcrypt unregistered [ 1037.047170] Key type ._llcrypt unregistered [ 1055.403449] Key type ._llcrypt registered [ 1055.405056] Key type .llcrypt registered [ 1055.447771] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 1055.924438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1055.936766] alg: No test for adler32 (adler32-zlib) [ 1056.832646] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 1056.935917] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 1058.527194] Key type lgssc registered [ 1058.938643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1060.278455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1060.286208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1061.387997] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1061.397111] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1061.462367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1061.876647] Lustre: Failing over lustre-MDT0000 [ 1061.984483] Lustre: server umount lustre-MDT0000 complete [ 1064.593638] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 07:31:33 (1788867093) [ 1067.542804] hrtimer: interrupt took 2918072 ns [ 1067.633995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1067.842110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1069.722455] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1073.101506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1073.166178] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1073.182393] Lustre: 64384:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1073.302558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1074.964401] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1078.129964] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1080.244846] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1082.936157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1083.038916] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1083.119788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1085.397978] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1088.419880] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1090.076407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1097.115813] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1099.232450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1099.235477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1099.240903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1100.257090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1100.257186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1100.262753] Lustre: Skipped 1 previous similar message [ 1100.265714] Lustre: Skipped 1 previous similar message [ 1105.311224] Lustre: server umount lustre-MDT0000 complete [ 1106.772152] LustreError: 66240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867137 with bad export cookie 4340291497323830839 [ 1106.775055] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1106.776634] LustreError: 66240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1106.896955] Lustre: server umount lustre-MDT0001 complete [ 1108.213944] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1109.984713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1113.093422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1116.639905] LustreError: 63307:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f75fe5f8 x1875763094651776/t0(0) o250->MGC192.168.206.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1116.758991] LustreError: 66233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1116.766312] LustreError: 66233:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1116.795393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1116.839160] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1118.345844] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1121.294550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1121.397282] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1122.954793] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1125.692688] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1126.495114] Lustre: 63311:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788867141/real 1788867141] req@00000000db108258 x1875763094651392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788867157 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1126.887840] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1127.567146] Lustre: 63309:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788867142/real 1788867142] req@00000000532f8604 x1875763094651520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788867158 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1127.772885] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1130.593676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1130.595245] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1130.595713] Lustre: Skipped 1 previous similar message [ 1130.599515] Lustre: Skipped 1 previous similar message [ 1131.615150] Lustre: 63309:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788867146/real 1788867146] req@00000000c9037f43 x1875763094651648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788867162 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1135.711204] Lustre: server umount lustre-MDT0000 complete [ 1136.097194] LustreError: 68152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1136.102231] LustreError: 68152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1137.233161] LustreError: 64377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867168 with bad export cookie 4340291497323833107 [ 1137.236853] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1137.237558] LustreError: 64377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1137.360526] Lustre: server umount lustre-MDT0001 complete [ 1139.910608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1143.686041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1147.359730] LustreError: 63307:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003484c74b x1875763094670976/t0(0) o250->MGC192.168.206.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1147.489010] LustreError: 66233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1147.519211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1147.523276] Lustre: Skipped 1 previous similar message [ 1147.556584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1149.133774] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1152.617381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1152.735376] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1154.307240] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1157.325967] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1157.855154] Lustre: 63308:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788867172/real 1788867172] req@000000006da76dcf x1875763094670592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788867188 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1157.868456] Lustre: 63308:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1158.116048] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1159.741892] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1162.207094] Lustre: 63311:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788867177/real 1788867177] req@00000000941b3d8a x1875763094670848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788867193 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1162.220886] Lustre: 63311:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1163.233941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1163.233983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1163.234943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1163.241786] Lustre: Skipped 1 previous similar message [ 1168.483437] Lustre: server umount lustre-MDT0000 complete [ 1170.002843] LustreError: 64377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867200 with bad export cookie 4340291497323834479 [ 1170.006500] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1170.008777] LustreError: 64377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1170.119326] Lustre: server umount lustre-MDT0001 complete [ 1172.700580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1174.181403] Lustre: DEBUG MARKER: files compared the same [ 1184.288851] Lustre: server umount lustre-OST0000 complete [ 1190.519667] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 1191.696687] Key type lgssc unregistered [ 1191.848461] LNet: 74099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1191.851529] LNetError: 74099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1191.864392] LNet: Removed LNI 192.168.206.152@tcp [ 1192.199925] Key type .llcrypt unregistered [ 1192.201758] Key type ._llcrypt unregistered [ 1202.200774] Key type ._llcrypt registered [ 1202.202197] Key type .llcrypt registered [ 1202.244636] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 1202.650359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1202.698514] alg: No test for adler32 (adler32-zlib) [ 1203.573909] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 1203.681863] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 1205.279138] Key type lgssc registered [ 1205.728380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1208.867543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1208.877701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1210.109262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1211.930020] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug +malloc all [ 1215.098539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1215.226802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1216.900085] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug +malloc all [ 1219.960316] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1222.262133] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1225.107611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1225.240060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1227.821360] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug +malloc all [ 1230.714860] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1234.469247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1239.521668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1239.525481] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1239.531865] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1239.535067] Lustre: Skipped 1 previous similar message [ 1240.934571] Lustre: server umount lustre-OST0000 complete [ 1244.639830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1244.640499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1244.646164] Lustre: Skipped 2 previous similar messages [ 1244.862668] Lustre: server umount lustre-MDT0000 complete [ 1246.484139] LustreError: 75471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867277 with bad export cookie 15907724635226894346 [ 1246.486420] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1246.493980] LustreError: 75471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1246.629266] Lustre: server umount lustre-MDT0001 complete [ 1251.117371] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 1252.326404] Key type lgssc unregistered [ 1252.470559] LNet: 79015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1252.473966] LNetError: 79015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1252.483552] LNet: Removed LNI 192.168.206.152@tcp [ 1252.844460] Key type .llcrypt unregistered [ 1252.846383] Key type ._llcrypt unregistered [ 1262.800206] Key type ._llcrypt registered [ 1262.801684] Key type .llcrypt registered [ 1262.843428] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 1263.198499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1263.242829] alg: No test for adler32 (adler32-zlib) [ 1264.129328] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 1264.226771] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 1265.823147] Key type lgssc registered [ 1266.223918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1269.361221] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1269.370267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1284.831347] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1289.951423] LustreError: 80379:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.152@tcp: failed processing log, type 4: rc = -110 [ 1316.575165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1318.922248] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1321.723560] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1324.679619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1324.739224] Lustre: *** cfs_fail_loc=706, val=0*** [ 1324.912160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1324.959908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1326.613938] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1330.127275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1330.149345] LustreError: 80405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1330.163390] LustreError: 80405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1330.360281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1331.986446] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1334.902275] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1337.237977] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1340.384735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1340.389816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1340.398144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1340.896750] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1340.901229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1340.909493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1341.923572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1341.926936] Lustre: Skipped 1 previous similar message [ 1344.289104] Lustre: server umount lustre-OST0000 complete [ 1347.040995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1347.045157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1347.052663] Lustre: Skipped 1 previous similar message [ 1347.062463] Lustre: Skipped 1 previous similar message [ 1352.161311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1352.165463] Lustre: Skipped 1 previous similar message [ 1352.286652] Lustre: server umount lustre-MDT0000 complete [ 1353.918461] LustreError: 80385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867384 with bad export cookie 15041657583821483298 [ 1353.920865] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1353.929014] LustreError: 80385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1354.056581] Lustre: server umount lustre-MDT0001 complete [ 1358.512426] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 1359.753743] Key type lgssc unregistered [ 1359.894530] LNet: 83988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1359.898323] LNetError: 83988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1359.908284] LNet: Removed LNI 192.168.206.152@tcp [ 1360.251171] Key type .llcrypt unregistered [ 1360.252612] Key type ._llcrypt unregistered [ 1371.066484] Key type ._llcrypt registered [ 1371.068159] Key type .llcrypt registered [ 1371.116165] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 1371.539067] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1371.572683] alg: No test for adler32 (adler32-zlib) [ 1372.436740] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 1372.545519] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 1374.135218] Key type lgssc registered [ 1374.594129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1377.724845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1377.738157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1380.323132] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1383.429155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1383.677595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1385.465538] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1388.742823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1388.852625] LustreError: 86064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1388.859530] LustreError: 86064:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1388.859730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1391.050099] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1394.018454] LustreError: 86068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1394.027025] LustreError: 86068:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1394.336099] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1394.342025] Lustre: MGS: Client 53e105a9-f4b4-4279-bcba-7c13a0d853ad (at 0@lo) reconnecting [ 1394.350915] LustreError: 86078:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1394.437746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1394.469921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1395.909853] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1404.897389] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1404.898771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1404.900561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1404.908680] Lustre: Skipped 1 previous similar message [ 1409.883290] Lustre: server umount lustre-OST0000 complete [ 1415.138746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1415.140189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1415.140251] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1415.147128] Lustre: Skipped 1 previous similar message [ 1415.154527] Lustre: Skipped 2 previous similar messages [ 1417.606875] Lustre: server umount lustre-MDT0000 complete [ 1419.415784] Lustre: server umount lustre-MDT0001 complete [ 1420.878286] Lustre: server umount MGS complete [ 1425.461879] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 07:37:34 (1788867454) [ 1435.923651] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 1439.321447] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 1453.150745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1454.723022] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1458.939956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1459.052628] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1459.089240] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1459.131473] Lustre: lustre-MDT0001: new disk, initializing [ 1459.164859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1459.175752] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1459.179758] Lustre: 93219:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1459.182510] LustreError: 93219:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1460.713558] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1465.260605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1465.359705] Lustre: lustre-OST0000: new disk, initializing [ 1465.361469] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1465.364435] Lustre: 93988:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1465.388133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1467.522923] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1468.832375] Lustre: 94367:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000009e0c27c9 x1875763426001792/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788867515 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1471.909327] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1474.719606] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1474.725252] Lustre: MGS: Client b0685414-785a-4567-a550-3246940bbda0 (at 0@lo) reconnecting [ 1474.740159] Lustre: 95034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1474.749202] LustreError: 93205:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1474.752714] LustreError: 93205:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1474.758317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1474.796721] Lustre: lustre-MDT0000: new disk, initializing [ 1474.855930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1474.867813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1475.935687] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1475.938173] Lustre: 95049:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000005437a1de x1875763426007936/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788867522 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1475.952018] Lustre: 95049:0:(client.c:1652:after_reply()) Skipped 1 previous similar message [ 1476.483620] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1485.026824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1485.031349] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1485.035149] Lustre: Skipped 1 previous similar message [ 1485.039016] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1509.856073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1509.859608] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1509.866087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1512.446742] Lustre: server umount lustre-OST0000 complete [ 1515.488060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1515.490417] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1515.495540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1515.502024] Lustre: Skipped 2 previous similar messages [ 1515.507277] Lustre: Skipped 3 previous similar messages [ 1520.446353] Lustre: server umount lustre-MDT0000 complete [ 1520.610333] LustreError: 95470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1520.617730] LustreError: 95470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1528.724123] Lustre: server umount lustre-MDT0001 complete [ 1530.557477] Lustre: server umount MGS complete [ 1536.228981] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 07:39:25 (1788867565) [ 1543.965364] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 1545.118180] Key type lgssc unregistered [ 1545.260318] LNet: 97550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1545.263878] LNetError: 97550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1545.274466] LNet: Removed LNI 192.168.206.152@tcp [ 1545.630478] Key type .llcrypt unregistered [ 1545.632359] Key type ._llcrypt unregistered [ 1554.436834] Key type ._llcrypt registered [ 1554.439184] Key type .llcrypt registered [ 1554.496814] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 1554.885945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1554.922072] alg: No test for adler32 (adler32-zlib) [ 1555.808147] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 1555.917927] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 1557.519209] Key type lgssc registered [ 1558.058188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1561.510787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1561.520021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1562.721778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1564.514838] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1568.111426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1568.280772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1569.949824] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1572.999346] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1575.182315] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1577.791530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1577.942829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1580.158768] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1583.168889] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1587.175094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1592.289910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1592.293229] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1592.293836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1592.298040] Lustre: Skipped 1 previous similar message [ 1593.562405] Lustre: server umount lustre-OST0000 complete [ 1597.321952] Lustre: server umount lustre-MDT0000 complete [ 1597.411308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1597.425241] LustreError: 98741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1597.433044] LustreError: 98741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1599.091611] LustreError: 98725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867629 with bad export cookie 12151559387160445640 [ 1599.094158] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1599.099788] LustreError: 98725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1599.248413] Lustre: server umount lustre-MDT0001 complete [ 1603.924210] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 1605.212090] Key type lgssc unregistered [ 1605.369571] LNet: 102277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1605.373259] LNetError: 102277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1605.385218] LNet: Removed LNI 192.168.206.152@tcp [ 1605.770608] Key type .llcrypt unregistered [ 1605.772238] Key type ._llcrypt unregistered [ 1613.544375] Key type ._llcrypt registered [ 1613.545807] Key type .llcrypt registered [ 1613.599789] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 1614.049352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1614.076489] alg: No test for adler32 (adler32-zlib) [ 1614.989593] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 1615.105680] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 1616.703156] Key type lgssc registered [ 1617.172650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1619.807281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1619.811652] LustreError: 103333:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 1619.811863] LustreError: 103334:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 1619.820083] LustreError: 103333:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 1619.824365] LustreError: 103333:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1619.829276] LustreError: 103333:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1619.836631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1621.041061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1624.391354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1624.525138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1626.282136] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1628.610567] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing lsmod [ 1629.854613] LustreError: 104533:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 1629.867420] LustreError: 104537:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 1629.871416] LustreError: 104533:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4986 [ 1629.874344] LustreError: 104533:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1629.879205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1629.881433] LustreError: 104533:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1630.075587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1631.078779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1634.401681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1634.403409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1634.408869] Lustre: Skipped 1 previous similar message [ 1637.490524] Lustre: server umount lustre-MDT0000 complete [ 1639.132120] LustreError: 103349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867669 with bad export cookie 6370346386809187739 [ 1639.134936] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1639.136684] LustreError: 103349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1639.904119] LustreError: 103366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1639.904338] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1639.914579] LustreError: 103366:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1639.919267] Lustre: Skipped 1 previous similar message [ 1639.919897] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1639.927879] Lustre: Skipped 2 previous similar messages [ 1640.543218] LustreError: 103800:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000078b0c073 x1875763679741440/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1640.611377] Lustre: server umount lustre-MDT0001 complete [ 1642.261327] Lustre: server umount lustre-OST0000 complete [ 1645.903877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1646.132149] LustreError: 105600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1646.165671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1647.937605] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1651.168185] LustreError: 105601:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1651.711658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1651.869413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1653.692496] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1656.976360] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1659.574345] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1662.804297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1662.983692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1665.615272] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1669.069502] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1670.186667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 1685.474814] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1685.480430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1685.493126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1686.503419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1691.621523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1691.624817] Lustre: Skipped 3 previous similar messages [ 1696.737657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1696.740600] Lustre: Skipped 1 previous similar message [ 1697.247116] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1697.306064] Lustre: server umount lustre-OST0000 complete [ 1701.858135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1701.863637] Lustre: Skipped 2 previous similar messages [ 1706.976203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1706.980082] Lustre: Skipped 3 previous similar messages [ 1713.119205] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1713.226691] Lustre: server umount lustre-MDT0000 complete [ 1715.235965] LustreError: 105580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867746 with bad export cookie 6370346386809188642 [ 1715.240823] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1715.243292] LustreError: 105580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1715.392687] Lustre: server umount lustre-MDT0001 complete [ 1721.465996] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 1722.857164] Key type lgssc unregistered [ 1723.020693] LNet: 109151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1723.029204] LNetError: 109151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1723.037773] LNet: Removed LNI 192.168.206.152@tcp [ 1723.433918] Key type .llcrypt unregistered [ 1723.435774] Key type ._llcrypt unregistered [ 1735.970871] Key type ._llcrypt registered [ 1735.972183] Key type .llcrypt registered [ 1736.020133] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 1736.520988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1736.554700] alg: No test for adler32 (adler32-zlib) [ 1737.468124] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 1737.601712] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 1739.191138] Key type lgssc registered [ 1739.733886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1743.728437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1743.740845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1744.944112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1746.873236] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1751.103364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1751.306496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1753.182916] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1756.769180] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1759.463751] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1762.858888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1763.026929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1765.979452] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1769.193945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 1769.787790] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1778.187868] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1779.859836] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1781.729287] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1781.730674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1781.738501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1781.744744] Lustre: Skipped 1 previous similar message [ 1781.750552] Lustre: Skipped 1 previous similar message [ 1786.851758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1786.857404] Lustre: Skipped 1 previous similar message [ 1791.968175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1796.063134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1796.147972] Lustre: server umount lustre-OST0000 complete [ 1802.209113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1802.210286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1802.214067] Lustre: Skipped 1 previous similar message [ 1802.216838] Lustre: Skipped 1 previous similar message [ 1804.081828] Lustre: server umount lustre-MDT0000 complete [ 1805.945787] LustreError: 110525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867836 with bad export cookie 17052786658720967599 [ 1805.949308] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1805.958782] LustreError: 110525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1806.119194] Lustre: server umount lustre-MDT0001 complete [ 1811.220473] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 1812.521741] Key type lgssc unregistered [ 1812.668748] LNet: 114178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1812.674474] LNetError: 114178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1812.685842] LNet: Removed LNI 192.168.206.152@tcp [ 1813.097301] Key type .llcrypt unregistered [ 1813.099110] Key type ._llcrypt unregistered [ 1822.028743] Key type ._llcrypt registered [ 1822.029947] Key type .llcrypt registered [ 1822.070501] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 1822.477898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1822.562614] alg: No test for adler32 (adler32-zlib) [ 1823.469247] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 1823.583468] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 1825.175856] Key type lgssc registered [ 1825.702663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1829.529968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1829.543418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1830.777930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1832.791606] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1837.274510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1837.500434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1839.814376] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1843.672569] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1846.504290] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1849.798081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1849.999733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1852.797157] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1856.170274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 1856.778319] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1866.720277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1866.727675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1866.737510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1867.745253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1867.749949] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1867.749974] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1868.073734] Lustre: server umount lustre-OST0000 complete [ 1872.865676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1872.877483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1872.881186] Lustre: Skipped 2 previous similar messages [ 1876.395615] Lustre: server umount lustre-MDT0000 complete [ 1877.984592] LustreError: 115374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1877.994918] LustreError: 115374:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1878.666330] LustreError: 115353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788867909 with bad export cookie 4673874381290053036 [ 1878.669874] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1878.671824] LustreError: 115353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1878.845488] Lustre: server umount lustre-MDT0001 complete [ 1884.901405] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 1886.410436] Key type lgssc unregistered [ 1886.628854] LNet: 118902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1886.632744] LNetError: 118902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1887.660577] LNet: Removed LNI 192.168.206.152@tcp [ 1888.190343] Key type .llcrypt unregistered [ 1888.192520] Key type ._llcrypt unregistered [ 1901.046449] Key type ._llcrypt registered [ 1901.048015] Key type .llcrypt registered [ 1901.106885] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 1901.490758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1901.525512] alg: No test for adler32 (adler32-zlib) [ 1902.426460] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 1902.560586] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 1904.167438] Key type lgssc registered [ 1904.730726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1908.445495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1908.455680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1909.671977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1911.809297] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1916.295530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1916.489460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1918.432959] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1921.723063] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1924.404392] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1927.630909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1927.823850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1930.956450] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 1934.615740] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1935.027298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 1948.247657] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1950.325540] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1950.328833] Lustre: Skipped 2 previous similar messages [ 1952.641656] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1952.646276] Lustre: Skipped 2 previous similar messages [ 1956.350886] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1956.363402] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1961.950097] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.52@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1961.954090] Lustre: Skipped 2 previous similar messages [ 1964.616925] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.52@tcp 192.168.0.[2,10]@tcp [ 1967.719777] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.52/24@tcp in log lustre-MDT0000 [ 1967.722956] Lustre: Skipped 2 previous similar messages [ 1973.767599] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.52/24@tcp [ 1973.771049] Lustre: Skipped 1 previous similar message [ 1976.289938] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1976.293059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1976.299287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1978.337550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1978.338793] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1978.344960] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1981.659250] Lustre: server umount lustre-OST0000 complete [ 1983.457156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.459823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1983.465892] Lustre: Skipped 1 previous similar message [ 1983.475276] Lustre: Skipped 1 previous similar message [ 1988.575960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1988.580865] Lustre: Skipped 1 previous similar message [ 1989.479638] Lustre: server umount lustre-MDT0000 complete [ 1991.144033] LustreError: 122940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868021 with bad export cookie 16028691158219405327 [ 1991.145115] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1991.150266] LustreError: 122940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1991.307836] Lustre: server umount lustre-MDT0001 complete [ 1995.941156] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 1997.183080] Key type lgssc unregistered [ 1997.322455] LNet: 125288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1997.325570] LNetError: 125288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1997.336559] LNet: Removed LNI 192.168.206.152@tcp [ 1997.705948] Key type .llcrypt unregistered [ 1997.707779] Key type ._llcrypt unregistered [ 2010.252574] Key type ._llcrypt registered [ 2010.253736] Key type .llcrypt registered [ 2010.298893] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 2010.689083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2010.749423] alg: No test for adler32 (adler32-zlib) [ 2011.629642] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 2011.740357] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 2013.335165] Key type lgssc registered [ 2013.839080] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2017.133206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2017.142381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2018.368091] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2018.376977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2018.382037] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.52/24@tcp [ 2020.164368] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2023.991403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2024.150147] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2024.163144] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.52/24@tcp [ 2024.185372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2026.194801] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2029.581217] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2031.970714] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2035.075143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2035.232961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2038.054948] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2041.389986] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2043.434610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 2053.602039] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2053.602335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2053.605927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2053.605986] LustreError: Skipped 1 previous similar message [ 2053.617291] Lustre: Skipped 1 previous similar message [ 2054.432274] Lustre: server umount lustre-OST0000 complete [ 2058.720686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2058.721447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2058.725275] Lustre: Skipped 1 previous similar message [ 2058.732125] Lustre: Skipped 2 previous similar messages [ 2062.326211] Lustre: server umount lustre-MDT0000 complete [ 2063.840286] LustreError: 127567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2063.849468] LustreError: 127567:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2063.924468] LustreError: 126803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868094 with bad export cookie 14434213729136476318 [ 2063.928246] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2063.930924] LustreError: 126803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2064.049858] Lustre: server umount lustre-MDT0001 complete [ 2068.314269] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 2069.463591] Key type lgssc unregistered [ 2069.599648] LNet: 130408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2069.604873] LNetError: 130408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2069.612642] LNet: Removed LNI 192.168.206.152@tcp [ 2070.010436] Key type .llcrypt unregistered [ 2070.012056] Key type ._llcrypt unregistered [ 2080.710530] Key type ._llcrypt registered [ 2080.712192] Key type .llcrypt registered [ 2080.759768] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 2088.010538] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 2088.398274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2088.471501] alg: No test for adler32 (adler32-zlib) [ 2089.339740] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 2089.439766] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 2091.031253] Key type lgssc registered [ 2091.512141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2107.508920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2107.521368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2108.648205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2108.664521] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2108.716331] Lustre: lustre-MDT0000: new disk, initializing [ 2108.756738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2108.771366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2110.685521] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2115.924509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2115.986582] Lustre: 133775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2116.019855] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2116.022961] Lustre: Skipped 1 previous similar message [ 2116.067383] Lustre: lustre-MDT0001: new disk, initializing [ 2116.117794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2116.137301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2116.141725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2118.125748] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2122.827457] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2125.453492] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2128.732291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2128.864117] Lustre: lustre-OST0000: new disk, initializing [ 2128.870549] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2128.876149] Lustre: 135744:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2128.917171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2131.576546] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2131.587463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2131.594048] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2131.650840] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2136.472453] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2138.375359] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2138.467307] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2140.191474] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2140.287593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2142.181051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2142.181669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2142.184473] LustreError: Skipped 1 previous similar message [ 2142.190619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2142.196596] Lustre: Skipped 1 previous similar message [ 2147.246866] Lustre: server umount lustre-OST0000 complete [ 2151.904323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2151.909721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2151.915818] Lustre: Skipped 1 previous similar message [ 2151.918672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2155.301893] Lustre: server umount lustre-MDT0000 complete [ 2156.862166] LustreError: 133767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868187 with bad export cookie 15882345363592108505 [ 2156.865085] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2156.867419] LustreError: 133767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2157.005558] Lustre: server umount lustre-MDT0001 complete [ 2160.953946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2161.175767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2162.836970] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2166.241565] LustreError: 137961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2166.250212] LustreError: 137961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2166.569183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2166.771178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2169.034129] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2172.882539] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2175.679663] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2179.193733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2179.380339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2182.047634] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2185.366932] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2187.331995] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2187.452844] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2189.470760] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2192.661237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 2197.025723] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2197.112041] Lustre: lustre-OST0001: new disk, initializing [ 2197.114222] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2197.117098] Lustre: 141349:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2197.163918] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2199.795319] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2204.029444] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2207.243249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2207.249853] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2207.294687] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2208.236704] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 2210.108603] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2210.211566] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2212.530217] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ccd06a0c000.ost_server_uuid 50 [ 2213.159724] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9ccd06a0c000.ost_server_uuid in FULL state after 0 sec [ 2217.440379] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2217.443337] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2217.448097] Lustre: Skipped 1 previous similar message [ 2217.450414] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2217.453946] Lustre: Skipped 3 previous similar messages [ 2221.837410] Lustre: server umount lustre-OST0001 complete [ 2222.573717] LustreError: 139783:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2222.580979] LustreError: 139783:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2226.656846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2226.661636] LustreError: Skipped 1 previous similar message [ 2226.664289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2226.672063] Lustre: Skipped 1 previous similar message [ 2226.674669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2227.682929] LustreError: 139783:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2227.695883] LustreError: 139783:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2229.538749] Lustre: server umount lustre-OST0000 complete [ 2232.800952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2232.803272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2232.805521] Lustre: Skipped 2 previous similar messages [ 2232.809821] Lustre: Skipped 3 previous similar messages [ 2237.417771] Lustre: server umount lustre-MDT0000 complete [ 2237.920118] LustreError: 137960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2237.928774] LustreError: 137960:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2238.984559] LustreError: 137986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868269 with bad export cookie 15882345363592109387 [ 2238.986071] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2238.991036] LustreError: 137986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2239.139166] Lustre: server umount lustre-MDT0001 complete [ 2247.400966] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 2248.805755] Key type lgssc unregistered [ 2248.967921] LNet: 144375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2248.970946] LNetError: 144375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2248.980061] LNet: Removed LNI 192.168.206.152@tcp [ 2249.418355] Key type .llcrypt unregistered [ 2249.419929] Key type ._llcrypt unregistered [ 2269.671348] Key type ._llcrypt registered [ 2269.673495] Key type .llcrypt registered [ 2269.735548] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 2277.609623] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 2278.220210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2278.236090] alg: No test for adler32 (adler32-zlib) [ 2279.156896] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 2279.317889] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 2280.959278] Key type lgssc registered [ 2281.520187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2296.865805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2296.876458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2298.012209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2298.029186] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2298.074936] Lustre: lustre-MDT0000: new disk, initializing [ 2298.149652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2298.162802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2300.014924] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2305.628750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2305.696819] Lustre: 148496:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2305.739259] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2305.743457] Lustre: Skipped 1 previous similar message [ 2305.789893] Lustre: lustre-MDT0001: new disk, initializing [ 2305.851042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2305.874165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2305.879500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2307.750512] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2312.473923] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2314.923842] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2318.152480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2318.284580] Lustre: lustre-OST0000: new disk, initializing [ 2318.286815] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2318.290209] Lustre: 150462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2318.336902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2321.303188] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2326.730732] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2328.078888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2328.086370] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2328.151261] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2334.973810] Lustre: Failing over lustre-OST0000 [ 2335.042253] Lustre: server umount lustre-OST0000 complete [ 2348.740839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2348.934546] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2348.954440] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2348.955129] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2351.404454] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2353.346989] Lustre: Failing over lustre-MDT0000 [ 2353.513123] Lustre: server umount lustre-MDT0000 complete [ 2354.146323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2354.154354] Lustre: Skipped 3 previous similar messages [ 2355.167799] Lustre: 146723:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788868369/real 1788868369] req@00000000854b1dd3 x1875764377056000/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788868385 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2359.178349] LustreError: 148502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2359.186650] LustreError: 148502:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2359.263254] Lustre: 146722:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788868374/real 1788868374] req@0000000050f60ac6 x1875764377058048/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788868390 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2364.299689] LustreError: 150936:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2364.309165] LustreError: 150936:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2364.383211] Lustre: 146723:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788868379/real 1788868379] req@00000000757fde7a x1875764377059200/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788868395 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2367.535884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2367.624142] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2367.630301] Lustre: 148499:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.152@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2367.732052] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2367.757983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2367.793882] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2369.184529] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2369.644151] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2373.096529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2373.104352] Lustre: lustre-OST0000: Recovery over after 0:25, of 3 clients 3 recovered and 0 were evicted. [ 2373.135049] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2378.208853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2378.209354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2378.213134] LustreError: Skipped 1 previous similar message [ 2378.213740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2378.228249] Lustre: Skipped 1 previous similar message [ 2380.716296] Lustre: server umount lustre-OST0000 complete [ 2383.328373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2383.330275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2383.337584] Lustre: Skipped 1 previous similar message [ 2383.344674] Lustre: Skipped 2 previous similar messages [ 2388.448157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2388.450608] Lustre: Skipped 1 previous similar message [ 2388.796197] Lustre: server umount lustre-MDT0000 complete [ 2390.740962] LustreError: 149292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868421 with bad export cookie 9640077396476666171 [ 2390.745720] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2390.747109] LustreError: 149292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2390.915356] Lustre: server umount lustre-MDT0001 complete [ 2396.452433] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 2397.757830] Key type lgssc unregistered [ 2397.912630] LNet: 154111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2397.916772] LNetError: 154111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2397.926136] LNet: Removed LNI 192.168.206.152@tcp [ 2398.325454] Key type .llcrypt unregistered [ 2398.326925] Key type ._llcrypt unregistered [ 2421.377309] Key type ._llcrypt registered [ 2421.378864] Key type .llcrypt registered [ 2421.431333] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 2421.951698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2421.963372] alg: No test for adler32 (adler32-zlib) [ 2422.854890] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 2422.978095] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 2424.583179] Key type lgssc registered [ 2425.110704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2428.635582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2428.645068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2429.867291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2431.834148] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2436.592485] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2438.931803] LustreError: 155654:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2438.936889] LustreError: 155654:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2438.941098] LustreError: 155654:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 2439.086852] Lustre: server umount lustre-MDT0000 complete [ 2445.585700] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 2449.559931] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 2466.446908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2466.558257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2466.578138] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2466.624187] Lustre: lustre-MDT0000: new disk, initializing [ 2466.665648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2466.675648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2468.574870] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2474.008425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2474.075423] Lustre: 158952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2474.119114] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2474.122919] Lustre: Skipped 1 previous similar message [ 2474.184368] Lustre: lustre-MDT0001: new disk, initializing [ 2474.240578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2474.264982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2474.271179] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2476.246526] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2481.315737] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2484.011319] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2487.345591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2487.497682] Lustre: lustre-OST0000: new disk, initializing [ 2487.501910] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2487.508269] Lustre: 160918:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2487.545971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2490.125726] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2494.716527] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2495.488526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2495.493145] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2495.533508] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2497.920571] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2498.645505] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 07:55:27 (1788868527) [ 2607.584043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2607.587960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2607.594794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2608.095524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2608.096528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2608.099499] Lustre: Skipped 1 previous similar message [ 2608.106826] Lustre: Skipped 1 previous similar message [ 2613.216367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2613.223740] Lustre: Skipped 3 previous similar messages [ 2613.487222] Lustre: server umount lustre-MDT0000 complete [ 2615.065772] LustreError: 158944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868645 with bad export cookie 15813327615833808665 [ 2615.066402] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2615.071601] LustreError: 158944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2618.336520] LustreError: 162679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2618.336599] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2618.344131] LustreError: 162679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2618.347887] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2620.383570] LustreError: 158960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2621.235763] Lustre: server umount lustre-MDT0001 complete [ 2637.279576] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2637.354466] Lustre: server umount lustre-OST0000 complete [ 2640.918108] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 2644.491303] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 2659.785143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2659.891740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2659.907556] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2659.947521] Lustre: lustre-MDT0000: new disk, initializing [ 2659.997934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2660.014622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2661.721440] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2666.567730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2666.614765] Lustre: 165550:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2666.642417] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2666.644680] Lustre: Skipped 1 previous similar message [ 2666.684635] Lustre: lustre-MDT0001: new disk, initializing [ 2666.741064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2666.744932] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2668.338647] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2672.851994] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2675.438236] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2678.423257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2678.544771] Lustre: lustre-OST0000: new disk, initializing [ 2678.546937] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2678.550489] Lustre: 167519:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2678.589064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2678.593316] Lustre: Skipped 1 previous similar message [ 2681.447759] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2685.877238] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2685.961658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2685.980345] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2686.019872] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2687.758110] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2687.851483] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2689.738645] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2689.837749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2691.043155] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2691.043671] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2691.046483] LustreError: Skipped 1 previous similar message [ 2691.053461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2691.058579] Lustre: Skipped 2 previous similar messages [ 2692.821016] Lustre: server umount lustre-OST0000 complete [ 2696.161443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2696.170148] Lustre: Skipped 2 previous similar messages [ 2700.601473] Lustre: server umount lustre-MDT0000 complete [ 2701.279845] LustreError: 166712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2701.287429] LustreError: 166712:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2702.314480] LustreError: 165542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868733 with bad export cookie 15813327615833967481 [ 2702.317582] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2702.321608] LustreError: 165542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2702.499763] Lustre: server umount lustre-MDT0001 complete [ 2706.338328] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:58:55 (1788868735) [ 2713.104996] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 2717.233932] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 2732.076583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2732.187611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2732.199835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2732.238971] Lustre: lustre-MDT0000: new disk, initializing [ 2732.279382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2732.286901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2734.141192] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2739.594178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2739.660561] Lustre: 172263:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2739.703514] LustreError: 172276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2739.819899] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2741.645783] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2746.303675] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2748.642833] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2751.815756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2751.962944] Lustre: 174243:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2754.644602] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2757.124525] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2757.180261] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2759.596452] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2767.330148] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2767.330307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2767.338301] LustreError: Skipped 1 previous similar message [ 2767.338966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2767.343619] Lustre: Skipped 1 previous similar message [ 2767.345328] Lustre: Skipped 2 previous similar messages [ 2768.479387] Lustre: server umount lustre-OST0000 complete [ 2770.400191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2776.307209] Lustre: server umount lustre-MDT0000 complete [ 2777.568415] LustreError: 173997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2777.576472] LustreError: 173997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2777.941953] LustreError: 172257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868808 with bad export cookie 15813327615833968349 [ 2777.944767] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2777.950960] LustreError: 172257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2778.084514] Lustre: server umount lustre-MDT0001 complete [ 2781.757788] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 08:00:10 (1788868810) [ 2788.654613] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 2792.490480] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 2808.498744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2808.630951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2808.653574] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2808.656927] Lustre: Skipped 3 previous similar messages [ 2808.700269] Lustre: lustre-MDT0000: new disk, initializing [ 2808.702290] Lustre: Skipped 2 previous similar messages [ 2808.733353] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2808.735993] Lustre: Skipped 2 previous similar messages [ 2808.743201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2808.746574] Lustre: Skipped 2 previous similar messages [ 2810.485564] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2815.431812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2815.481934] Lustre: 178556:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2815.600623] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2817.222179] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2821.826544] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2824.147287] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2827.227862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2827.355466] Lustre: 180531:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2828.349760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2830.107494] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2834.647143] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2838.496531] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2838.498598] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2838.500209] LustreError: Skipped 1 previous similar message [ 2838.500690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2838.507591] Lustre: Skipped 3 previous similar messages [ 2838.509270] Lustre: Skipped 4 previous similar messages [ 2844.200778] Lustre: server umount lustre-OST0000 complete [ 2852.078896] Lustre: server umount lustre-MDT0000 complete [ 2853.860105] LustreError: 178567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2853.870764] LustreError: 178567:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2853.923278] LustreError: 178549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868884 with bad export cookie 15813327615833969434 [ 2853.927032] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2853.928905] LustreError: 178549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2854.074384] Lustre: server umount lustre-MDT0001 complete [ 2858.865721] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 2860.131856] Key type lgssc unregistered [ 2860.272558] LNet: 182396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2860.277349] LNetError: 182396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2860.286424] LNet: Removed LNI 192.168.206.152@tcp [ 2860.669755] Key type .llcrypt unregistered [ 2860.672255] Key type ._llcrypt unregistered [ 2871.695895] Key type ._llcrypt registered [ 2871.697644] Key type .llcrypt registered [ 2871.749246] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 2872.290728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2872.307968] alg: No test for adler32 (adler32-zlib) [ 2873.213284] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 2873.341946] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 2874.951204] Key type lgssc registered [ 2875.522774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2879.144269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2879.157583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2880.362323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2882.210331] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2885.876539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2886.045651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2887.814393] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2891.032399] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2893.988741] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2897.297170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2897.473730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2900.314166] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2903.753466] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2906.735644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2926.560976] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2926.561751] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2926.569579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2931.681845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2931.684598] Lustre: Skipped 1 previous similar message [ 2932.319634] Lustre: server umount lustre-OST0000 complete [ 2936.800909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2936.801713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2936.807427] Lustre: Skipped 2 previous similar messages [ 2940.076161] Lustre: server umount lustre-MDT0000 complete [ 2941.581459] LustreError: 184516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788868972 with bad export cookie 47035434366553538 [ 2941.584582] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2941.586477] LustreError: 184516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2941.725973] Lustre: server umount lustre-MDT0001 complete [ 2945.858517] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 2947.070401] Key type lgssc unregistered [ 2947.224813] LNet: 187332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2947.227920] LNetError: 187332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2947.236494] LNet: Removed LNI 192.168.206.152@tcp [ 2947.596356] Key type .llcrypt unregistered [ 2947.597789] Key type ._llcrypt unregistered [ 2957.926577] Key type ._llcrypt registered [ 2957.929191] Key type .llcrypt registered [ 2957.975696] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 2958.360974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2958.429706] alg: No test for adler32 (adler32-zlib) [ 2959.341702] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 2959.474108] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 2961.087121] Key type lgssc registered [ 2961.647781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2965.139299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2965.151912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2966.373490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2968.185421] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2971.869470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2972.054414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2973.820432] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2977.162473] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2979.433168] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2982.272558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2982.484966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2984.955627] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 2988.160712] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2992.744105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3002.338473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3002.343234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3002.345111] Lustre: Skipped 1 previous similar message [ 3002.350641] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3004.821632] Lustre: lustre-OST0000: Not available for connect from 192.168.206.52@tcp (stopping) [ 3004.824960] Lustre: Skipped 1 previous similar message [ 3006.933865] Lustre: server umount lustre-OST0000 complete [ 3009.788388] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9ccd04e76800.ost_server_uuid 50 [ 3010.405699] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9ccd04e76800.ost_server_uuid in DISCONN state after 0 sec [ 3011.121270] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3021.211387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.212159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3021.217162] Lustre: Skipped 1 previous similar message [ 3027.357736] Lustre: server umount lustre-MDT0000 complete [ 3027.424060] LustreError: 188723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3027.432483] LustreError: 188723:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3028.777400] LustreError: 188704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869059 with bad export cookie 13906890007894625196 [ 3028.780706] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3028.782557] LustreError: 188704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3028.921829] Lustre: server umount lustre-MDT0001 complete [ 3032.293672] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 08:04:21 (1788869061) [ 3034.950243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3035.167882] LustreError: 192680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3035.206581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3036.908909] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3040.224509] LustreError: 192681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3040.392950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3040.559311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3042.366278] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3045.244762] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3047.508086] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3050.413986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3050.591121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3053.158341] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3056.383377] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3059.280435] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3059.374960] Lustre: lustre-OST0001: new disk, initializing [ 3059.377553] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3059.380766] Lustre: 195518:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3060.454827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3060.625971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3060.632175] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3060.678335] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3061.789961] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3066.046671] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3067.361801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3067.364788] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3067.371362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3067.373914] Lustre: Skipped 3 previous similar messages [ 3070.948831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3070.955875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3071.384220] Lustre: lustre-OST0000: Not available for connect from 192.168.206.52@tcp (stopping) [ 3071.388745] Lustre: Skipped 2 previous similar messages [ 3073.166126] Lustre: server umount lustre-OST0000 complete [ 3075.574103] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3075.663811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3076.070722] LustreError: 194500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3076.081760] LustreError: 194500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3077.526650] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3077.622743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3081.189813] LustreError: 194642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3081.196722] LustreError: 194642:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3091.426096] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3091.426669] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3091.430540] LustreError: 194497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3091.430551] LustreError: 194497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3091.435247] LustreError: Skipped 1 previous similar message [ 3091.443672] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3091.462881] Lustre: Skipped 1 previous similar message [ 3094.440191] Lustre: server umount lustre-OST0001 complete [ 3102.581252] Lustre: server umount lustre-MDT0000 complete [ 3104.211922] LustreError: 192662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869135 with bad export cookie 13906890007894626253 [ 3104.214088] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3104.217737] LustreError: 192662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3104.353748] Lustre: server umount lustre-MDT0001 complete [ 3113.913978] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 08:05:43 (1788869143) [ 3116.480566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3116.606321] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3116.619858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3116.711037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3116.717557] Lustre: Skipped 1 previous similar message [ 3118.442121] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3122.050316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3122.121887] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3122.145849] Lustre: 198932:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3123.980779] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3126.931467] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3129.321942] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3132.313548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3132.404853] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3134.952688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3135.172875] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3138.382324] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3141.485308] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3141.530509] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3144.308920] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3147.488856] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3149.827326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3149.828976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3149.835159] Lustre: Skipped 4 previous similar messages [ 3149.844387] Lustre: Skipped 5 previous similar messages [ 3163.103128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3163.152184] Lustre: server umount lustre-OST0000 complete [ 3165.155084] LustreError: 202562:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3165.167400] LustreError: 202562:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3175.395974] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3175.397480] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3175.408303] LustreError: Skipped 1 previous similar message [ 3180.719353] Lustre: server umount lustre-OST0001 complete [ 3183.584684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3183.588838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3183.591126] Lustre: Skipped 11 previous similar messages [ 3188.768611] Lustre: server umount lustre-MDT0000 complete [ 3190.440768] LustreError: 198924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869221 with bad export cookie 13906890007894627611 [ 3190.445041] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3190.447781] LustreError: 198924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3190.628774] Lustre: server umount lustre-MDT0001 complete [ 3202.763530] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 08:07:11 (1788869231) [ 3211.430327] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 3216.050423] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 3233.617234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3233.731469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3233.748691] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3233.794099] Lustre: lustre-MDT0000: new disk, initializing [ 3233.839442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3233.842824] Lustre: Skipped 3 previous similar messages [ 3233.855061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3235.701278] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3241.052126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3241.113324] Lustre: 206993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3241.151057] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3241.153884] Lustre: Skipped 1 previous similar message [ 3241.191682] Lustre: lustre-MDT0001: new disk, initializing [ 3241.250397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3241.254900] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3243.267321] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3248.217186] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3251.003155] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3254.513975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3254.653141] Lustre: lustre-OST0000: new disk, initializing [ 3254.658500] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3254.661756] Lustre: 208959:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3257.221114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3257.226057] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3257.284931] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3257.516550] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3261.946132] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3263.883607] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3263.980713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3265.692742] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3265.778178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3267.554079] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3267.554764] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3267.559862] Lustre: Skipped 3 previous similar messages [ 3267.561302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3267.566044] LustreError: Skipped 1 previous similar message [ 3267.568507] Lustre: Skipped 3 previous similar messages [ 3272.676105] Lustre: server umount lustre-OST0000 complete [ 3280.563863] Lustre: server umount lustre-MDT0000 complete [ 3282.162076] LustreError: 207786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869312 with bad export cookie 13906890007894629116 [ 3282.165388] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3282.167931] LustreError: 207786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3282.315128] Lustre: server umount lustre-MDT0001 complete [ 3285.651442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3285.817577] LustreError: 211171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3285.827812] LustreError: 211171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 3287.395809] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3290.994835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3293.008477] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3296.467318] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3299.167280] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3302.496701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3302.662651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3302.666753] Lustre: Skipped 4 previous similar messages [ 3305.300633] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3308.655311] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3310.734419] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3311.898315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3313.879100] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3317.061734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 3324.066605] Lustre: server umount lustre-OST0000 complete [ 3326.657609] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3328.429717] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3353.097083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3356.243131] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3359.396568] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3359.401268] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3359.401808] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3359.535124] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3360.192226] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3364.833605] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3364.835386] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3364.836249] LustreError: Skipped 1 previous similar message [ 3364.845163] Lustre: Skipped 6 previous similar messages [ 3367.772052] Lustre: server umount lustre-OST0000 complete [ 3375.733352] Lustre: server umount lustre-MDT0000 complete [ 3377.754325] LustreError: 211152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869408 with bad export cookie 13906890007894629998 [ 3377.756339] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3377.758703] LustreError: 211152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3377.976939] Lustre: server umount lustre-MDT0001 complete [ 3382.359763] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 08:10:11 (1788869411) [ 3385.604428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3385.868431] LustreError: 217159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3387.796198] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3391.813490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3394.036570] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3397.135427] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3400.013701] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3403.555880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3406.751586] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3410.279561] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3412.147192] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3412.257264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3414.160508] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3417.320865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 3420.279366] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3420.363961] Lustre: lustre-OST0001: new disk, initializing [ 3420.366720] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3420.371128] Lustre: 220546:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3423.180333] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3428.201621] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3428.881924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3428.886240] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3428.943161] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3430.278821] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3430.389447] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3432.223718] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3432.322571] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3433.959641] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3433.961411] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3433.966067] LustreError: Skipped 2 previous similar messages [ 3433.970750] Lustre: Skipped 13 previous similar messages [ 3435.341786] Lustre: server umount lustre-OST0001 complete [ 3438.142225] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3438.248503] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3439.984787] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3440.073416] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3464.147399] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3464.261042] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3464.264970] Lustre: Skipped 5 previous similar messages [ 3466.492760] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3469.284964] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3469.295718] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3469.298684] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3469.310558] Lustre: Skipped 2 previous similar messages [ 3469.710740] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3470.371990] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3477.329968] Lustre: server umount lustre-OST0001 complete [ 3493.856689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3493.861542] Lustre: Skipped 7 previous similar messages [ 3496.418158] Lustre: server umount lustre-OST0000 complete [ 3504.290834] Lustre: server umount lustre-MDT0000 complete [ 3505.970840] LustreError: 217893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869536 with bad export cookie 13906890007894630950 [ 3505.974320] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3505.975966] LustreError: 217893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3512.208938] Lustre: server umount lustre-MDT0001 complete [ 3523.801894] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 08:12:32 (1788869552) [ 3526.754805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3526.898484] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3526.919443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3528.666253] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3532.212228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3532.285303] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3532.299247] Lustre: 225780:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3534.293916] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3537.696375] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3540.335924] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3543.435586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3543.536168] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3546.250770] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3549.936140] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3555.070966] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3555.117184] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3558.878516] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3563.079361] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3563.960196] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3565.887625] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3565.986509] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3568.348044] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3568.498648] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3571.691420] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ccd04e73800.ost_server_uuid 50 [ 3572.577617] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9ccd04e73800.ost_server_uuid in FULL state after 0 sec [ 3573.124403] Lustre: Permanently deactivating lustre-OST0001 [ 3573.133175] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3574.353025] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 3574.717835] Lustre: Permanently reactivating lustre-OST0001 [ 3582.465303] Lustre: server umount lustre-OST0001 complete [ 3585.504789] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3585.509772] LustreError: Skipped 6 previous similar messages [ 3586.019986] LustreError: 227620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3586.028025] LustreError: 227620:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [ 3590.370692] Lustre: server umount lustre-OST0000 complete [ 3601.375194] LustreError: 230345:0:(osp_dev.c:699:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 3602.447753] Lustre: server umount lustre-MDT0000 complete [ 3604.358216] LustreError: 227637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869635 with bad export cookie 13906890007894632238 [ 3604.361868] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3604.363787] LustreError: 227637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3604.521521] Lustre: server umount lustre-MDT0001 complete [ 3615.864166] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 08:14:04 (1788869644) [ 3619.325412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3619.459838] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3619.477663] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3619.483386] Lustre: Skipped 5 previous similar messages [ 3621.175898] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3624.879041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3624.933335] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3624.948043] Lustre: 232469:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3626.706764] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3629.747698] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3632.042426] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3634.926242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3635.049610] Lustre: Permanently deactivating lustre-OST0000 [ 3635.057081] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3637.648962] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3640.752859] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3641.850959] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3641.854196] Lustre: Skipped 1 previous similar message [ 3643.907638] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3646.580724] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3647.843672] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3649.718776] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3650.447797] Lustre: Permanently reactivating lustre-OST0000 [ 3655.164354] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3655.168337] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3655.173507] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3655.195301] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3655.202703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3668.758431] Lustre: server umount lustre-OST0001 complete [ 3677.286894] Lustre: server umount lustre-OST0000 complete [ 3685.288857] Lustre: server umount lustre-MDT0000 complete [ 3686.869780] LustreError: 235342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869717 with bad export cookie 13906890007894633967 [ 3686.873584] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3686.875610] LustreError: 235342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3687.033909] Lustre: server umount lustre-MDT0001 complete [ 3691.731365] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 3692.982111] Key type lgssc unregistered [ 3693.122789] LNet: 237341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3693.125947] LNetError: 237341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3693.133485] LNet: Removed LNI 192.168.206.152@tcp [ 3693.497916] Key type .llcrypt unregistered [ 3693.499364] Key type ._llcrypt unregistered [ 3704.334737] Key type ._llcrypt registered [ 3704.336445] Key type .llcrypt registered [ 3704.382774] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 3704.882562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3704.900703] alg: No test for adler32 (adler32-zlib) [ 3705.780628] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 3705.887745] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 3707.479233] Key type lgssc registered [ 3707.962865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3712.257528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3712.266875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3713.468171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3715.156239] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3718.767976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3718.833359] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3718.847547] Lustre: 238770:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3718.853461] Lustre: Permanently deactivating lustre-MDT0001 [ 3718.859905] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3718.960196] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3718.964550] LustreError: 238813:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3718.970899] LustreError: 238813:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3718.979817] LustreError: 238813:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 1, failed: rc = -108 [ 3719.004175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3720.772644] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3724.038100] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3726.531297] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3729.583775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3729.748231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3732.405054] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3735.581322] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3738.607806] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3738.732095] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3739.751091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3739.752231] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3741.158451] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3743.848738] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3743.852047] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 3744.105978] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3745.839114] Lustre: Permanently reactivating lustre-MDT0001 [ 3745.846395] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3745.849214] Lustre: Skipped 2 previous similar messages [ 3751.317973] Lustre: lustre-MDT0001: Client 10ff46d0-d6bf-443d-bfbe-87f600ae8731 (at 192.168.206.52@tcp) reconnecting [ 3754.535183] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3755.207422] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3757.434717] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3757.531635] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3759.357985] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3759.460441] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3760.645463] Lustre: Permanently deactivating lustre-MDT0001 [ 3760.652192] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3760.659810] Lustre: Skipped 2 previous similar messages [ 3766.293285] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3769.824664] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3769.834075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3769.836425] Lustre: Skipped 3 previous similar messages [ 3769.839693] Lustre: Skipped 2 previous similar messages [ 3774.947975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3774.951845] Lustre: Skipped 3 previous similar messages [ 3775.480617] Lustre: server umount lustre-MDT0000 complete [ 3777.039388] LustreError: 238763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869807 with bad export cookie 2510040331140073103 [ 3777.040752] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3777.045074] LustreError: 238763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3777.174624] Lustre: server umount lustre-MDT0001 complete [ 3784.761954] Lustre: server umount lustre-OST0000 complete [ 3788.978485] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 08:16:58 (1788869818) [ 3793.128775] Lustre: server umount lustre-OST0001 complete [ 3795.655145] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 3799.516159] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 3814.720080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3814.825370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3814.830753] Lustre: Skipped 2 previous similar messages [ 3814.843156] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3814.884987] Lustre: lustre-MDT0000: new disk, initializing [ 3814.917732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3814.928234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3816.861697] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3821.947152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3822.008542] Lustre: 247075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3822.039374] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3822.043264] Lustre: Skipped 1 previous similar message [ 3822.087888] Lustre: lustre-MDT0001: new disk, initializing [ 3822.144095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3822.148290] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3823.875618] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3828.464795] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3830.655206] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3833.602097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3833.729878] Lustre: lustre-OST0000: new disk, initializing [ 3833.732465] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3833.735719] Lustre: 249042:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3833.780023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3833.784493] Lustre: Skipped 1 previous similar message [ 3836.395885] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3840.762593] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3842.570257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3842.575484] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3842.617768] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3846.699762] LustreError: 249050:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3851.605501] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3851.688349] Lustre: lustre-OST0001: new disk, initializing [ 3851.691211] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3851.695710] Lustre: 250154:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3851.738688] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3854.274971] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3856.791114] LustreError: 249050:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3856.802341] LustreError: 247082:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3859.008808] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3860.509150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3860.516879] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3860.567588] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3860.828899] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 3866.895113] LustreError: 247082:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3870.110474] Lustre: lustre-OST0001: Not available for connect from 192.168.206.52@tcp (stopping) [ 3870.696641] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3870.700330] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3873.938312] Lustre: server umount lustre-OST0001 complete [ 3875.212780] LustreError: 249023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3875.221942] LustreError: 249023:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3875.811553] LustreError: 249025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3875.819745] LustreError: 249025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3880.932709] LustreError: 249024:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3880.941321] LustreError: 249024:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3886.051534] LustreError: 249759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3886.059334] LustreError: 249759:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3888.608476] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3888.609283] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3888.613267] LustreError: Skipped 2 previous similar messages [ 3888.619503] Lustre: Skipped 1 previous similar message [ 3888.622481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3888.626204] Lustre: Skipped 3 previous similar messages [ 3891.170531] LustreError: 249025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3891.180986] LustreError: 249025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3893.022812] Lustre: server umount lustre-OST0000 complete [ 3896.289341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3896.292188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3896.296479] Lustre: Skipped 1 previous similar message [ 3896.299210] Lustre: Skipped 2 previous similar messages [ 3900.899952] Lustre: server umount lustre-MDT0000 complete [ 3901.408246] LustreError: 247085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3901.414963] LustreError: 247085:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3902.481148] LustreError: 247870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788869933 with bad export cookie 2510040331140076148 [ 3902.485090] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3902.485974] LustreError: 247870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3908.676521] Lustre: server umount lustre-MDT0001 complete [ 3913.243281] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 3914.491470] Key type lgssc unregistered [ 3914.636689] LNet: 252183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3914.641547] LNetError: 252183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3914.651176] LNet: Removed LNI 192.168.206.152@tcp [ 3915.020108] Key type .llcrypt unregistered [ 3915.021829] Key type ._llcrypt unregistered [ 3932.849543] Key type ._llcrypt registered [ 3932.851397] Key type .llcrypt registered [ 3932.899102] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 3933.366775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3933.383600] alg: No test for adler32 (adler32-zlib) [ 3934.277324] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 3934.382357] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 3935.983160] Key type lgssc registered [ 3936.416178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3939.716968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3939.730824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3940.848182] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3940.856830] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3940.936073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3942.654431] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3946.654506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3946.724648] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3946.740045] Lustre: 254364:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3946.866517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3948.744670] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3952.020729] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3954.530157] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3957.365309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3957.473855] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3957.551650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3960.197240] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 3963.319366] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3967.515361] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4013.538768] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4013.539327] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4013.544570] LustreError: Skipped 1 previous similar message [ 4013.551898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4013.555453] Lustre: Skipped 1 previous similar message [ 4014.853580] Lustre: server umount lustre-OST0000 complete [ 4016.771090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4021.390551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4021.510231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4023.793206] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4026.767787] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4033.763460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4033.770744] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4033.770812] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4033.780632] Lustre: Skipped 1 previous similar message [ 4064.739296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4064.740097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4064.744412] Lustre: Skipped 1 previous similar message [ 4064.745579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4064.751586] Lustre: Skipped 1 previous similar message [ 4067.187108] Lustre: server umount lustre-OST0000 complete [ 4069.857915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4069.858191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4069.858923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4069.862452] Lustre: Skipped 2 previous similar messages [ 4069.865360] LustreError: Skipped 1 previous similar message [ 4074.848516] Lustre: server umount lustre-MDT0000 complete [ 4076.597349] LustreError: 255103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870107 with bad export cookie 4802175191713882291 [ 4076.600412] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4076.603425] LustreError: 255103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4076.751928] Lustre: server umount lustre-MDT0001 complete [ 4081.235575] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 4082.440170] Key type lgssc unregistered [ 4082.616817] LNet: 259780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4082.621867] LNetError: 259780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4082.631736] LNet: Removed LNI 192.168.206.152@tcp [ 4083.037974] Key type .llcrypt unregistered [ 4083.040088] Key type ._llcrypt unregistered [ 4093.682747] Key type ._llcrypt registered [ 4093.684333] Key type .llcrypt registered [ 4093.730277] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 4094.175847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4094.216395] alg: No test for adler32 (adler32-zlib) [ 4095.113618] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 4095.234930] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 4096.831152] Key type lgssc registered [ 4097.247972] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4100.134693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4100.141526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4101.300619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4102.842451] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4106.180691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4106.378135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4107.863563] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4110.583898] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4112.672447] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4115.094166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4115.208897] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4117.154602] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4119.557157] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4126.505232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4142.052529] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4142.052834] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4142.059582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4142.065971] Lustre: Skipped 1 previous similar message [ 4144.736805] Lustre: server umount lustre-OST0000 complete [ 4147.169034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4147.169634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4147.175485] Lustre: Skipped 2 previous similar messages [ 4147.179922] Lustre: Skipped 1 previous similar message [ 4152.288189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4152.290142] Lustre: Skipped 1 previous similar message [ 4152.408828] Lustre: server umount lustre-MDT0000 complete [ 4153.645327] LustreError: 261150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870184 with bad export cookie 6072432804478272942 [ 4153.647527] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4153.649870] LustreError: 261150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4153.766816] Lustre: server umount lustre-MDT0001 complete [ 4157.536286] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 4158.638387] Key type lgssc unregistered [ 4158.780538] LNet: 265490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4158.784362] LNetError: 265490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4158.798401] LNet: Removed LNI 192.168.206.152@tcp [ 4159.160229] Key type .llcrypt unregistered [ 4159.161757] Key type ._llcrypt unregistered [ 4166.966157] Key type ._llcrypt registered [ 4166.967458] Key type .llcrypt registered [ 4167.017867] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 4167.394701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4167.460892] alg: No test for adler32 (adler32-zlib) [ 4168.332610] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 4168.429853] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 4170.015184] Key type lgssc registered [ 4170.455250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4173.597557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4173.607839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4174.786256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4176.169837] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4179.014026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4179.143349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4180.512481] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4182.949830] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4184.793616] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4187.064615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4187.180860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4189.291410] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4192.012319] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4199.466570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4208.608073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4208.611120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4208.617217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4209.633612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4209.635147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4209.635650] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4212.314035] Lustre: server umount lustre-OST0000 complete [ 4214.755556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4214.756534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4214.760767] Lustre: Skipped 1 previous similar message [ 4214.767534] Lustre: Skipped 2 previous similar messages [ 4219.875605] LustreError: 266685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4219.883839] LustreError: 266685:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4219.885732] Lustre: server umount lustre-MDT0000 complete [ 4221.370242] LustreError: 266665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870252 with bad export cookie 11005155605323893410 [ 4221.373343] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4221.375121] LustreError: 266665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4221.497965] Lustre: server umount lustre-MDT0001 complete [ 4225.715497] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 4226.762053] Key type lgssc unregistered [ 4226.891428] LNet: 270381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4226.894420] LNetError: 270381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4226.902428] LNet: Removed LNI 192.168.206.152@tcp [ 4227.236323] Key type .llcrypt unregistered [ 4227.237866] Key type ._llcrypt unregistered [ 4236.940476] Key type ._llcrypt registered [ 4236.942152] Key type .llcrypt registered [ 4236.986935] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 4237.367146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4237.399059] alg: No test for adler32 (adler32-zlib) [ 4238.269303] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 4238.370087] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 4239.959176] Key type lgssc registered [ 4240.367827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4243.306476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4243.315038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4244.466182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4245.976475] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4248.944743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4249.083254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4250.585576] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4253.314660] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4255.223035] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4257.533789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4257.639573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4259.561361] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4262.070888] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4270.950629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4281.312217] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4281.314663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4281.315924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4281.320851] Lustre: Skipped 1 previous similar message [ 4286.433300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4286.436331] Lustre: Skipped 2 previous similar messages [ 4286.871317] Lustre: server umount lustre-OST0000 complete [ 4290.016131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4290.019412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4290.027034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4291.553270] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4294.434605] Lustre: server umount lustre-MDT0000 complete [ 4296.027449] LustreError: 271752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870326 with bad export cookie 4452887940873327304 [ 4296.031128] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4296.033941] LustreError: 271752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4296.175396] Lustre: server umount lustre-MDT0001 complete [ 4300.668105] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 4301.817933] Key type lgssc unregistered [ 4301.955463] LNet: 276092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4301.958550] LNetError: 276092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4301.965426] LNet: Removed LNI 192.168.206.152@tcp [ 4302.307451] Key type .llcrypt unregistered [ 4302.309047] Key type ._llcrypt unregistered [ 4310.006738] Key type ._llcrypt registered [ 4310.008209] Key type .llcrypt registered [ 4310.050220] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 4310.464149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4310.482974] alg: No test for adler32 (adler32-zlib) [ 4311.351586] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 4311.455977] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 4313.047146] Key type lgssc registered [ 4313.494295] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4316.529183] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4316.540574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4317.750988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4319.430487] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4322.870732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4323.055231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4324.754563] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4327.686053] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4329.876159] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4332.449537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4332.589097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4334.902967] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4337.679025] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4342.820924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4352.996269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4352.996340] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4352.997514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4353.000901] Lustre: Skipped 1 previous similar message [ 4353.003039] LustreError: Skipped 1 previous similar message [ 4357.851403] Lustre: server umount lustre-OST0000 complete [ 4363.232556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4363.233229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4363.238300] Lustre: Skipped 1 previous similar message [ 4363.240335] Lustre: Skipped 1 previous similar message [ 4365.339171] Lustre: server umount lustre-MDT0000 complete [ 4366.641296] LustreError: 277271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870397 with bad export cookie 5096482157081506895 [ 4366.644145] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4366.646217] LustreError: 277271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4366.772651] Lustre: server umount lustre-MDT0001 complete [ 4370.453488] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 4371.466798] Key type lgssc unregistered [ 4371.597383] LNet: 281021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4371.600122] LNetError: 281021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4371.607313] LNet: Removed LNI 192.168.206.152@tcp [ 4371.915489] Key type .llcrypt unregistered [ 4371.916524] Key type ._llcrypt unregistered [ 4375.685448] Key type ._llcrypt registered [ 4375.686807] Key type .llcrypt registered [ 4375.727407] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4381.665888] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 4388.100589] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 4388.605792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4388.613600] alg: No test for adler32 (adler32-zlib) [ 4389.476809] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 4389.581551] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 4391.167157] Key type lgssc registered [ 4391.528584] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4404.378083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4404.386868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4405.486343] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4405.497910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4405.538336] Lustre: lustre-MDT0000: new disk, initializing [ 4405.567276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4405.575607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4407.094151] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4411.590345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4411.637277] Lustre: 284539:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4411.661239] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4411.663097] Lustre: Skipped 1 previous similar message [ 4411.700430] Lustre: lustre-MDT0001: new disk, initializing [ 4411.741867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4411.761196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4411.765763] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4413.257141] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4417.539683] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4419.952228] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4422.902940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4423.028292] Lustre: lustre-OST0000: new disk, initializing [ 4423.030883] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4423.034465] Lustre: 286512:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4423.070926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4423.999916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4424.004533] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4424.055673] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4425.476134] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4429.954238] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4431.710523] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4431.798978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4433.392114] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4433.482131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4434.402618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4434.404588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4434.406117] LustreError: Skipped 1 previous similar message [ 4434.406907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4434.410423] Lustre: Skipped 1 previous similar message [ 4434.418047] Lustre: Skipped 1 previous similar message [ 4439.521029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4439.523726] Lustre: Skipped 1 previous similar message [ 4440.352055] Lustre: server umount lustre-OST0000 complete [ 4442.592832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4442.596635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4442.604684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4444.640987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4444.641592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4448.105711] Lustre: server umount lustre-MDT0000 complete [ 4449.645855] LustreError: 285338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870480 with bad export cookie 13948194115559419798 [ 4449.649898] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4449.654022] LustreError: 285338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4449.793041] Lustre: server umount lustre-MDT0001 complete [ 4453.372447] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 08:28:02 (1788870482) [ 4456.209935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4456.502557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4458.306735] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4461.537226] LustreError: 288919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4461.544258] LustreError: 288919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4462.207221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4462.405986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4464.241266] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4467.232551] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4469.268426] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4471.922249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4472.055579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4474.255993] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4477.137764] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4486.625333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4486.627309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4486.630832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4486.636654] Lustre: Skipped 1 previous similar message [ 4486.639624] Lustre: Skipped 1 previous similar message [ 4491.621739] Lustre: server umount lustre-OST0000 complete [ 4493.280464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4493.284184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4496.865949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4496.871531] Lustre: Skipped 3 previous similar messages [ 4499.471575] Lustre: server umount lustre-MDT0000 complete [ 4501.364243] LustreError: 291497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870532 with bad export cookie 13948194115559420666 [ 4501.364733] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4501.365108] LustreError: 289654:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4501.365228] LustreError: 289654:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000057d28d9e x1875766589583616/t0(0) o1000->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 264/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4501.371345] LustreError: 291497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4501.513309] Lustre: server umount lustre-MDT0001 complete [ 4505.867136] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 4507.042404] Key type lgssc unregistered [ 4507.180488] LNet: 292448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4507.183264] LNetError: 292448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4507.193376] LNet: Removed LNI 192.168.206.152@tcp [ 4507.546108] Key type .llcrypt unregistered [ 4507.547574] Key type ._llcrypt unregistered [ 4518.224348] Key type ._llcrypt registered [ 4518.225815] Key type .llcrypt registered [ 4518.273460] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 4525.289512] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 4525.685631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4525.714262] alg: No test for adler32 (adler32-zlib) [ 4526.601159] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 4526.705294] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 4528.311168] Key type lgssc registered [ 4528.790834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4548.152185] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4548.160916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4549.270112] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4549.287762] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4549.337196] Lustre: lustre-MDT0000: new disk, initializing [ 4549.370669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4549.381594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4550.999855] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4556.280168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4556.339168] Lustre: 296242:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4556.372139] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4556.374466] Lustre: Skipped 1 previous similar message [ 4556.414563] Lustre: lustre-MDT0001: new disk, initializing [ 4556.460637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4556.485458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4556.489874] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4558.063891] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4562.090075] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4564.282361] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4567.160043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4567.278509] Lustre: lustre-OST2710: new disk, initializing [ 4567.280622] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4567.284265] Lustre: 298207:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4567.331432] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4569.709477] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4573.713111] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4576.100237] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4576.174172] Lustre: lustre-OST03e8: new disk, initializing [ 4576.176403] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4576.179252] Lustre: 299260:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4576.200163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4576.203911] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4576.224239] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4576.281868] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4576.290388] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000400 [ 4578.218447] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4580.636141] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4583.663981] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4583.748832] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4585.309876] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4585.393463] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4586.932377] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4587.015026] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4588.584479] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4588.673065] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4591.588277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4591.590284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4591.598971] Lustre: Skipped 3 previous similar messages [ 4591.603107] Lustre: Skipped 3 previous similar messages [ 4595.895111] Lustre: server umount lustre-MDT0000 complete [ 4597.246779] LustreError: 296234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870628 with bad export cookie 572432300711528724 [ 4597.251040] LustreError: 296234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4597.251458] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4597.372833] Lustre: server umount lustre-MDT0001 complete [ 4609.027715] Lustre: server umount lustre-OST2710 complete [ 4620.663432] Lustre: server umount lustre-OST03e8 complete [ 4628.421475] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 4632.155789] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 4645.134898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4645.239983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4645.258661] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4645.307053] Lustre: lustre-MDT0000: new disk, initializing [ 4645.340210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4645.348721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4645.353039] Lustre: Skipped 1 previous similar message [ 4646.864322] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4651.118787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4651.169626] Lustre: 304509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4651.278366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4651.282352] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4651.285944] Lustre: Skipped 1 previous similar message [ 4652.755376] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4656.871739] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4658.959430] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4661.749382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4661.886096] Lustre: lustre-OST0000: new disk, initializing [ 4661.888248] Lustre: Skipped 1 previous similar message [ 4661.890761] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4661.893280] Lustre: Skipped 2 previous similar messages [ 4661.896746] Lustre: 306478:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4661.936514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4661.939100] Lustre: Skipped 1 previous similar message [ 4664.495450] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4668.420166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4668.425145] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4668.459176] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4668.811749] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4670.582800] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4670.679180] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4672.417807] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4672.510456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4673.507144] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4673.508843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4673.515328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4673.517508] LustreError: Skipped 1 previous similar message [ 4673.522783] Lustre: Skipped 1 previous similar message [ 4675.349681] Lustre: server umount lustre-OST0000 complete [ 4677.088191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4677.091131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4677.096284] Lustre: Skipped 1 previous similar message [ 4677.098915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4682.859298] Lustre: server umount lustre-MDT0000 complete [ 4683.744260] LustreError: 305321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4683.750903] LustreError: 305321:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4684.420749] LustreError: 304502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870715 with bad export cookie 572432300711530327 [ 4684.424419] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4684.426664] LustreError: 304502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4684.553528] Lustre: server umount lustre-MDT0001 complete [ 4687.953087] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 08:31:57 (1788870717) [ 4688.539695] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4689.188410] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 08:31:58 (1788870718) [ 4690.387120] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 4700.482791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4700.580145] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4700.593057] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4700.655558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4702.210747] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4705.334760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4705.400976] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4705.417224] Lustre: 310330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4707.035309] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4723.173241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4723.262935] Lustre: Denying initial registration attempt from nid 192.168.206.152@tcp, specified as failover [ 4723.268175] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4723.272305] LustreError: 311780:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 4723.275807] LustreError: 311780:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 4723.279308] LustreError: 311780:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4723.315385] Lustre: server umount lustre-OST0000 complete [ 4723.317116] LustreError: 311780:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4726.241825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4726.245245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4726.251155] Lustre: Skipped 1 previous similar message [ 4726.254115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4726.256983] Lustre: Skipped 2 previous similar messages [ 4732.061725] Lustre: server umount lustre-MDT0000 complete [ 4733.408360] LustreError: 310339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4733.414730] LustreError: 310339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4733.482497] LustreError: 310322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870764 with bad export cookie 572432300711531195 [ 4733.486844] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4733.488026] LustreError: 310322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4733.614332] Lustre: server umount lustre-MDT0001 complete [ 4737.538423] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 4738.795850] Key type lgssc unregistered [ 4738.946714] LNet: 312897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4738.950985] LNetError: 312897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4740.008712] LNet: Removed LNI 192.168.206.152@tcp [ 4740.434386] Key type .llcrypt unregistered [ 4740.436154] Key type ._llcrypt unregistered [ 4745.230226] Key type ._llcrypt registered [ 4745.231486] Key type .llcrypt registered [ 4745.276536] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 4745.670457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4745.693470] alg: No test for adler32 (adler32-zlib) [ 4746.567813] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 4746.674264] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 4748.263143] Key type lgssc registered [ 4748.747328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4774.846439] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 4778.251171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4778.259767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4779.393177] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4779.406841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4779.476285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4781.073429] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4784.364843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4784.440183] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4784.456276] Lustre: 315633:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4784.597642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4786.356130] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4789.420173] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4791.743927] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4807.778317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4807.857511] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4807.935926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4809.999794] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4812.573647] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4813.395383] Lustre: server umount lustre-OST0000 complete [ 4814.829774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4814.830705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4814.834084] Lustre: Skipped 1 previous similar message [ 4819.936417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4819.940416] Lustre: Skipped 2 previous similar messages [ 4820.827548] Lustre: server umount lustre-MDT0000 complete [ 4822.244291] LustreError: 315623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870853 with bad export cookie 7122707901387087119 [ 4822.247632] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4822.248957] LustreError: 315623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4822.373393] Lustre: server umount lustre-MDT0001 complete [ 4826.259126] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 4827.328816] Key type lgssc unregistered [ 4827.463347] LNet: 319193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4827.466360] LNetError: 319193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4827.479419] LNet: Removed LNI 192.168.206.152@tcp [ 4827.798760] Key type .llcrypt unregistered [ 4827.800307] Key type ._llcrypt unregistered [ 4837.539155] Key type ._llcrypt registered [ 4837.540658] Key type .llcrypt registered [ 4837.582142] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 4838.018429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4838.030899] alg: No test for adler32 (adler32-zlib) [ 4838.929305] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 4839.035993] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 4840.623200] Key type lgssc registered [ 4841.069964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4843.986283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4843.991825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4845.153559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4846.790117] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4850.309867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4850.465092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4851.962173] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4854.692101] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4856.557610] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4860.834439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4860.956443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4863.016149] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4865.650897] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4880.354072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4880.354775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4880.357856] Lustre: Skipped 2 previous similar messages [ 4880.360953] Lustre: Skipped 2 previous similar messages [ 4882.321254] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.52@tcp (stopping) [ 4885.472965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4885.475134] Lustre: Skipped 2 previous similar messages [ 4885.526622] Lustre: server umount lustre-MDT0000 complete [ 4886.759211] LustreError: 320564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870917 with bad export cookie 2224674736279081661 [ 4886.762149] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4886.764226] LustreError: 320564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4887.433597] LustreError: 320577:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4887.439764] LustreError: 320577:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4890.591935] LustreError: 320576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4890.592212] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4890.601073] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4890.603783] Lustre: Skipped 1 previous similar message [ 4891.616409] LustreError: 323071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4896.735929] LustreError: 320578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4896.736234] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4896.742199] LustreError: 320578:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4896.744192] Lustre: Skipped 2 previous similar messages [ 4901.343186] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4901.420674] Lustre: server umount lustre-MDT0001 complete [ 4903.638714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4906.784494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4912.095560] LustreError: 319892:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f99648a5 x1875767060357632/t0(0) o250->MGC192.168.206.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4912.226766] LustreError: 322411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4912.236945] LustreError: 322411:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4912.262448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4912.286375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4913.826341] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4917.262869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4917.417421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4918.990336] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4921.936286] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4926.949341] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4928.647279] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec [ 4930.663068] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4931.256075] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4933.092941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4933.095134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4933.096991] LustreError: Skipped 1 previous similar message [ 4933.103180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4933.107016] Lustre: Skipped 2 previous similar messages [ 4938.779384] Lustre: server umount lustre-OST0000 complete [ 4942.305376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4942.310756] Lustre: Skipped 1 previous similar message [ 4946.453367] Lustre: server umount lustre-MDT0000 complete [ 4947.424226] LustreError: 324198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4947.432817] LustreError: 324198:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4947.822289] LustreError: 320563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788870978 with bad export cookie 2224674736279088780 [ 4947.825721] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4947.827327] LustreError: 320563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4947.949676] Lustre: server umount lustre-MDT0001 complete [ 4951.661607] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 4952.725957] Key type lgssc unregistered [ 4952.864377] LNet: 326864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4952.866502] LNetError: 326864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4952.874346] LNet: Removed LNI 192.168.206.152@tcp [ 4953.209909] Key type .llcrypt unregistered [ 4953.211411] Key type ._llcrypt unregistered [ 4963.979069] Key type ._llcrypt registered [ 4963.980444] Key type .llcrypt registered [ 4964.026435] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 4964.365658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4964.392480] alg: No test for adler32 (adler32-zlib) [ 4965.276968] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 4965.381766] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 4966.975287] Key type lgssc registered [ 4967.441326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4970.409936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4970.419204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4971.619578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4973.191400] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4976.410130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4976.548905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4978.103508] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 4981.029910] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4983.282374] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4986.528992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4986.535266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4986.537739] Lustre: Skipped 1 previous similar message [ 4990.628110] Lustre: server umount lustre-MDT0000 complete [ 4991.968313] LustreError: 328270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4991.973842] LustreError: 328270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4992.091282] LustreError: 328251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871022 with bad export cookie 3368538233370330609 [ 4992.094532] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4992.097269] LustreError: 328251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4992.217304] Lustre: server umount lustre-MDT0001 complete [ 4998.367958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4998.509405] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4998.531845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4998.628265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5000.250031] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5003.636055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5003.700610] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5003.716545] Lustre: 330852:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5003.835458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5005.386588] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5008.357646] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5010.596797] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5015.842672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5015.911712] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5015.916735] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5015.920609] LustreError: 332734:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 5015.924360] LustreError: 332734:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 5015.927524] LustreError: 332734:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5015.959594] Lustre: server umount lustre-OST0000 complete [ 5015.961890] LustreError: 332734:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5021.030681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5021.119760] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5021.197444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5023.413336] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5026.263083] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5026.836379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 5029.181147] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5029.214988] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5029.249616] Lustre: lustre-OST0001: new disk, initializing [ 5029.251625] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5029.254824] Lustre: 334275:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5029.298449] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5031.520119] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5034.357325] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5035.516869] Lustre: server umount lustre-OST0001 complete [ 5036.548081] LustreError: 333200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5037.024356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5037.025453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5037.028834] LustreError: Skipped 1 previous similar message [ 5037.033356] Lustre: Skipped 1 previous similar message [ 5037.033943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5037.045420] Lustre: Skipped 1 previous similar message [ 5041.635574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5041.637598] LustreError: 333200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5041.637948] Lustre: Skipped 1 previous similar message [ 5041.643160] LustreError: 333200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5046.755297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5046.755994] LustreError: 335277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5046.758270] Lustre: Skipped 1 previous similar message [ 5046.765836] LustreError: 335277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5051.359129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5051.424680] Lustre: server umount lustre-OST0000 complete [ 5054.943928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5054.947490] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5054.951840] Lustre: Skipped 1 previous similar message [ 5054.954271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5056.992647] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5067.231147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5067.232707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5067.237064] Lustre: Skipped 5 previous similar messages [ 5067.287135] Lustre: server umount lustre-MDT0000 complete [ 5068.622459] LustreError: 330844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871099 with bad export cookie 3368538233370331148 [ 5068.625105] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5068.627589] LustreError: 330844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5068.744908] Lustre: server umount lustre-MDT0001 complete [ 5072.512361] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 5073.685133] Key type lgssc unregistered [ 5073.821404] LNet: 336229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5073.825277] LNetError: 336229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5073.835464] LNet: Removed LNI 192.168.206.152@tcp [ 5074.188553] Key type .llcrypt unregistered [ 5074.190244] Key type ._llcrypt unregistered [ 5107.744698] Key type ._llcrypt registered [ 5107.746571] Key type .llcrypt registered [ 5107.799440] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 5114.890885] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 5115.396786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5115.406088] alg: No test for adler32 (adler32-zlib) [ 5116.282274] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 5116.385370] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 5117.975165] Key type lgssc registered [ 5118.417474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5132.412642] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5132.424050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5133.529369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5133.542318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5133.579869] Lustre: lustre-MDT0000: new disk, initializing [ 5133.609492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5133.618578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5135.258823] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5139.928212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5139.982426] Lustre: 341125:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5140.010233] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5140.013029] Lustre: Skipped 1 previous similar message [ 5140.053587] Lustre: lustre-MDT0001: new disk, initializing [ 5140.108613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5140.131352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5140.136841] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5141.657895] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5145.614212] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5147.783726] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5150.385275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5150.505353] Lustre: lustre-OST0000: new disk, initializing [ 5150.508292] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5150.513079] Lustre: 343092:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5150.549163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5152.998309] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5157.689365] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5159.602527] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5160.971607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5160.976546] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5161.012994] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5161.773293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5163.811894] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5163.930415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5166.052312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5166.052824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5166.058571] LustreError: Skipped 1 previous similar message [ 5166.067905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5166.073740] Lustre: Skipped 1 previous similar message [ 5170.971127] Lustre: server umount lustre-OST0000 complete [ 5176.288628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5176.288936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5176.289767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5176.292834] Lustre: Skipped 2 previous similar messages [ 5178.718963] Lustre: server umount lustre-MDT0000 complete [ 5180.029187] LustreError: 341922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871210 with bad export cookie 14273240655873409061 [ 5180.032511] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5180.034473] LustreError: 341922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5180.144860] Lustre: server umount lustre-MDT0001 complete [ 5183.203808] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 08:40:12 (1788871212) [ 5185.826241] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 08:40:15 (1788871215) [ 5189.224321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5189.467504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5191.022300] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5194.210983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5194.349680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5195.965259] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5198.917166] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5201.111719] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5203.792575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5203.929405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5206.170289] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5209.000799] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5218.057376] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5218.724486] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5219.356549] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5220.014264] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5220.685335] Lustre: Failing over lustre-MDT0000 [ 5220.860638] Lustre: server umount lustre-MDT0000 complete [ 5224.417278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5224.420073] LustreError: 345855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5224.423574] Lustre: Skipped 2 previous similar messages [ 5224.428451] LustreError: 345855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5229.536132] LustreError: 345859:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5229.542432] LustreError: 345859:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5234.062114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5234.126168] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5234.246665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5234.283687] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5235.082388] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5235.830178] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5239.267730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5239.285826] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5239.309661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5241.532460] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5242.195995] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5244.244308] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5245.074554] Lustre: Failing over lustre-MDT0000 [ 5245.222828] Lustre: server umount lustre-MDT0000 complete [ 5245.331687] LustreError: 345855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.52@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5245.339783] LustreError: 345855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5249.505912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5249.507164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5249.510229] LustreError: 345860:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5249.514399] Lustre: Skipped 2 previous similar messages [ 5249.521015] LustreError: 345860:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5254.624336] LustreError: 346587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5254.630130] LustreError: 346587:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5258.872115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5258.933645] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5259.082508] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5260.637688] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5260.682621] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5264.359966] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5264.363695] Lustre: Skipped 2 previous similar messages [ 5264.380087] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5264.402053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5266.538607] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5267.151407] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5274.593134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5274.593162] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5274.595167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5274.595171] Lustre: Skipped 1 previous similar message [ 5274.598399] Lustre: Skipped 1 previous similar message [ 5275.879199] Lustre: server umount lustre-MDT0000 complete [ 5277.443251] LustreError: 345839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871308 with bad export cookie 14273240655873411756 [ 5277.444493] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5277.449039] LustreError: 345839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5277.570164] Lustre: server umount lustre-MDT0001 complete [ 5289.506671] Lustre: server umount lustre-OST0000 complete [ 5295.015832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5295.169710] LustreError: 352192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5295.176058] LustreError: 352192:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5295.194035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5295.196693] Lustre: Skipped 1 previous similar message [ 5296.731814] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5299.821920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5301.410899] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5303.820878] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5305.791962] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5308.154273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5310.363082] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5313.317881] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5321.637992] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5326.821874] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5326.822976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5326.825692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5326.830160] Lustre: Skipped 2 previous similar messages [ 5326.834916] Lustre: Skipped 3 previous similar messages [ 5327.125680] Lustre: server umount lustre-OST0000 complete [ 5330.731452] Lustre: server umount lustre-MDT0000 complete [ 5330.913055] LustreError: 352192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5332.340666] LustreError: 352172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871363 with bad export cookie 14273240655873412505 [ 5332.343741] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5332.345944] LustreError: 352172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5332.484141] Lustre: server umount lustre-MDT0001 complete [ 5336.591342] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 5337.703645] Key type lgssc unregistered [ 5337.858685] LNet: 355741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5337.861876] LNetError: 355741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5337.874482] LNet: Removed LNI 192.168.206.152@tcp [ 5338.215174] Key type .llcrypt unregistered [ 5338.216658] Key type ._llcrypt unregistered [ 5348.741600] Key type ._llcrypt registered [ 5348.743097] Key type .llcrypt registered [ 5348.785732] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 5349.155616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5349.171989] alg: No test for adler32 (adler32-zlib) [ 5350.046971] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 5350.151934] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 5351.735214] Key type lgssc registered [ 5352.144787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5355.013406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5355.023057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5356.208224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5357.790339] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5361.159573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5361.313299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5362.878428] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5365.653370] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5367.815404] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5370.202148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5370.353506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5372.498987] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5375.194843] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5381.606761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5391.840703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5391.843416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5391.845939] Lustre: Skipped 1 previous similar message [ 5395.673582] Lustre: server umount lustre-MDT0000 complete [ 5396.947196] LustreError: 357210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871427 with bad export cookie 3929459940931034784 [ 5396.950176] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5396.951175] LustreError: 357210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5396.960794] LustreError: 357225:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5396.960973] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5396.967691] LustreError: 357225:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5396.972117] Lustre: Skipped 1 previous similar message [ 5396.978112] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5396.980478] Lustre: Skipped 2 previous similar messages [ 5397.054649] Lustre: server umount lustre-MDT0001 complete [ 5404.455388] Lustre: server umount lustre-OST0000 complete [ 5409.512210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5409.688750] LustreError: 360955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5409.714531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5411.326497] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5414.722760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5414.877340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5416.493226] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5419.146404] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5421.293870] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5423.913934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5424.058253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5426.354208] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5429.192582] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5434.280799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5444.578973] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5444.580134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5444.584085] Lustre: Skipped 2 previous similar messages [ 5444.586886] Lustre: Skipped 1 previous similar message [ 5447.646944] Lustre: server umount lustre-MDT0000 complete [ 5449.070134] LustreError: 360937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871479 with bad export cookie 3929459940931035890 [ 5449.073766] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5449.076804] LustreError: 360937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5449.196554] Lustre: server umount lustre-MDT0001 complete [ 5460.960719] Lustre: server umount lustre-OST0000 complete [ 5467.076983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5467.257130] LustreError: 364864:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5467.281386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5468.831550] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5472.297602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5474.177523] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5477.059663] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5479.185610] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5481.809995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5484.047580] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5486.719276] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5492.201702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5502.433793] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5502.435700] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5502.440083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5502.444614] Lustre: Skipped 1 previous similar message [ 5504.743179] Lustre: server umount lustre-OST0000 complete [ 5507.553288] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5507.571349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5507.576215] Lustre: Skipped 2 previous similar messages [ 5512.483702] Lustre: server umount lustre-MDT0000 complete [ 5512.672238] LustreError: 365589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5512.678946] LustreError: 365589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5514.086854] LustreError: 364845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871544 with bad export cookie 3929459940931036912 [ 5514.090515] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5514.096046] LustreError: 364845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5514.249217] Lustre: server umount lustre-MDT0001 complete [ 5518.610657] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 5519.822487] Key type lgssc unregistered [ 5519.986794] LNet: 368409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5519.990944] LNetError: 368409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5520.001584] LNet: Removed LNI 192.168.206.152@tcp [ 5520.381157] Key type .llcrypt unregistered [ 5520.382785] Key type ._llcrypt unregistered [ 5532.800700] Key type ._llcrypt registered [ 5532.803362] Key type .llcrypt registered [ 5532.855369] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 5533.236850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5533.281359] alg: No test for adler32 (adler32-zlib) [ 5534.169681] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 5534.276690] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 5535.871135] Key type lgssc registered [ 5536.389997] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5539.802377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5539.821846] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5539.826273] LustreError: 369821:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5539.837882] LustreError: 369821:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5539.841863] LustreError: 369821:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5539.845410] LustreError: 369821:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5539.850421] LustreError: 369821:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5542.884797] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5542.889076] LustreError: 370217:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5542.900437] LustreError: 370217:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5542.903940] LustreError: 370217:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5542.907422] LustreError: 370217:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5542.912128] LustreError: 370217:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5550.884817] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 5552.009688] Key type lgssc unregistered [ 5552.161386] LNet: 371217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5552.165959] LNetError: 371217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5552.176473] LNet: Removed LNI 192.168.206.152@tcp [ 5552.507445] Key type .llcrypt unregistered [ 5552.509043] Key type ._llcrypt unregistered [ 5559.793337] Key type ._llcrypt registered [ 5559.795084] Key type .llcrypt registered [ 5559.846027] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 5566.479031] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 5566.990779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5567.016178] alg: No test for adler32 (adler32-zlib) [ 5567.882533] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 5567.977648] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 5569.575173] Key type lgssc registered [ 5569.986991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5582.865538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5582.871963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5583.960903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5583.976190] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5584.018294] Lustre: lustre-MDT0000: new disk, initializing [ 5584.049270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5584.057604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5585.592872] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5590.055420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5590.107266] Lustre: 374346:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5590.134736] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5590.137849] Lustre: Skipped 1 previous similar message [ 5590.176596] Lustre: lustre-MDT0001: new disk, initializing [ 5590.215169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5590.231350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5590.235872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5591.906225] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5596.321591] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5598.571876] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5601.311963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5601.433221] Lustre: lustre-OST0000: new disk, initializing [ 5601.436356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5601.440260] Lustre: 376324:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5601.477491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5603.966158] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5608.194814] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5609.821301] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5611.519274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5611.524392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5611.569948] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5611.941937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5613.512188] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5613.606414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5616.608604] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5616.609506] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5616.611802] LustreError: Skipped 1 previous similar message [ 5616.617490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5616.621327] Lustre: Skipped 1 previous similar message [ 5620.443444] Lustre: server umount lustre-OST0000 complete [ 5626.335929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5626.339889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5626.345790] Lustre: Skipped 1 previous similar message [ 5626.348254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5628.126570] Lustre: server umount lustre-MDT0000 complete [ 5629.525243] LustreError: 374338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871660 with bad export cookie 694893749327126530 [ 5629.528777] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5629.530689] LustreError: 374338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5629.656991] Lustre: server umount lustre-MDT0001 complete [ 5632.773799] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5633.385402] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 08:47:42 (1788871662) [ 5635.905322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5636.137796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5637.798246] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5641.155865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5641.184208] LustreError: 378794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5641.195542] LustreError: 378794:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5641.345589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5643.059539] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5646.031537] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5648.436495] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5651.196401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5651.343018] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5653.702669] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5656.573811] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5659.334845] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5659.416096] Lustre: lustre-OST0001: new disk, initializing [ 5659.419054] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5659.422469] Lustre: 381627:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5661.875151] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5665.815278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5665.819488] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5665.877063] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5666.499705] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5670.883849] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5670.887657] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5670.888178] LustreError: Skipped 1 previous similar message [ 5670.894258] Lustre: Skipped 1 previous similar message [ 5670.897182] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5670.902026] Lustre: Skipped 2 previous similar messages [ 5674.644987] Lustre: server umount lustre-OST0001 complete [ 5676.005461] LustreError: 380611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5676.012823] LustreError: 380611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5681.128379] LustreError: 380614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5681.135521] LustreError: 380614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5686.248088] LustreError: 380766:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5686.256317] LustreError: 380766:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5688.290933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5688.296085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5688.307967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5688.311694] Lustre: Skipped 1 previous similar message [ 5691.363021] LustreError: 380614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5691.371713] LustreError: 380614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5693.650813] Lustre: server umount lustre-OST0000 complete [ 5696.482126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5696.484204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5696.490273] Lustre: Skipped 2 previous similar messages [ 5696.493086] Lustre: Skipped 2 previous similar messages [ 5701.518422] Lustre: server umount lustre-MDT0000 complete [ 5701.601221] LustreError: 378794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5703.226312] LustreError: 378774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871734 with bad export cookie 694893749327127398 [ 5703.229746] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5703.233647] LustreError: 378774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5709.470674] Lustre: server umount lustre-MDT0001 complete [ 5714.254856] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 5715.670192] Key type lgssc unregistered [ 5715.830539] LNet: 383608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5715.834867] LNetError: 383608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5715.849384] LNet: Removed LNI 192.168.206.152@tcp [ 5716.301558] Key type .llcrypt unregistered [ 5716.303329] Key type ._llcrypt unregistered [ 5739.536994] Key type ._llcrypt registered [ 5739.538586] Key type .llcrypt registered [ 5739.591669] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 5740.010594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5740.058916] alg: No test for adler32 (adler32-zlib) [ 5740.960284] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 5741.061437] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 5742.657419] Key type lgssc registered [ 5743.087661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5746.030154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5746.038498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5747.125269] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5747.133902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5747.192101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5748.561685] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5751.548582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5751.588547] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5751.601207] Lustre: 386377:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5751.712776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5753.202897] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5756.000153] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5758.173980] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5762.016068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5762.020971] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5762.027969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5764.064208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5764.064837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5769.185253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5769.188087] Lustre: Skipped 2 previous similar messages [ 5773.279252] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5773.343676] Lustre: server umount lustre-MDT0000 complete [ 5774.713915] LustreError: 386371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871805 with bad export cookie 13107263276759384615 [ 5774.714692] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5774.718398] LustreError: 386371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5774.842658] Lustre: server umount lustre-MDT0001 complete [ 5778.634131] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 08:50:07 (1788871807) [ 5781.240103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5781.461235] LustreError: 388889:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5781.468961] LustreError: 388889:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5781.485046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5783.131290] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5786.532705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5786.592665] LustreError: 388890:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5786.705987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5788.592386] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5791.819986] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5794.363488] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5797.428526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5797.544911] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5797.642397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5800.327610] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5803.874436] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5814.906170] Lustre: Permanently deactivating lustre-OST0000 [ 5814.916442] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5815.223653] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5815.952360] LustreError: 391680:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5815.957524] LustreError: 391680:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 5815.960918] LustreError: 391680:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5817.226789] LustreError: 391728:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 5817.230876] LustreError: 391728:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5824.308249] Lustre: server umount lustre-OST0000 complete [ 5825.610270] LustreError: 391978:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 5825.612842] LustreError: 391978:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5827.552227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5827.556470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5827.565449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5833.185413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5833.189415] Lustre: Skipped 3 previous similar messages [ 5836.289989] Lustre: server umount lustre-MDT0000 complete [ 5837.927370] LustreError: 388869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871868 with bad export cookie 13107263276759385182 [ 5837.930698] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5837.934661] LustreError: 388869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5847.590567] Lustre: server umount lustre-MDT0001 complete [ 5851.032048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5852.671125] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5853.797441] LustreError: 393134:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5856.449773] Lustre: 393473:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.152@tcp [ 5857.259770] Lustre: server umount MGS complete [ 5862.074638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5862.291971] LustreError: 394196:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5862.298878] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5862.308145] Lustre: Skipped 1 previous similar message [ 5862.339113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5864.121339] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5867.488308] LustreError: 394197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5867.822362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5867.982571] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5869.847787] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5872.855156] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5875.040523] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5875.736813] Lustre: Permanently reactivating lustre-OST0000 [ 5875.744330] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5875.747045] Lustre: Skipped 2 previous similar messages [ 5897.695148] LustreError: 395862:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5900.919021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5901.069252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5901.071963] Lustre: Skipped 1 previous similar message [ 5903.419784] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5906.588545] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5922.788682] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5922.788827] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5922.792060] LustreError: Skipped 1 previous similar message [ 5922.796737] Lustre: Skipped 1 previous similar message [ 5922.799650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5922.804025] Lustre: Skipped 1 previous similar message [ 5924.567814] Lustre: server umount lustre-OST0000 complete [ 5927.905142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5927.912162] Lustre: Skipped 1 previous similar message [ 5932.131794] Lustre: server umount lustre-MDT0000 complete [ 5933.024273] LustreError: 394196:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5933.030884] LustreError: 394196:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5933.562113] LustreError: 395769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788871964 with bad export cookie 13107263276759386204 [ 5933.566933] LustreError: 395769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5933.567418] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5933.681780] Lustre: server umount lustre-MDT0001 complete [ 5937.581895] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 5938.736092] Key type lgssc unregistered [ 5938.880739] LNet: 398674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5938.883845] LNetError: 398674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5938.892403] LNet: Removed LNI 192.168.206.152@tcp [ 5939.266499] Key type .llcrypt unregistered [ 5939.268071] Key type ._llcrypt unregistered [ 5946.750890] Key type ._llcrypt registered [ 5946.752541] Key type .llcrypt registered [ 5946.797351] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 5953.887310] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 5954.244096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5954.281084] alg: No test for adler32 (adler32-zlib) [ 5955.166567] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 5955.280502] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 5956.871153] Key type lgssc registered [ 5957.366386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5972.147922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5972.158778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5973.254948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5973.268439] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5973.306468] Lustre: lustre-MDT0000: new disk, initializing [ 5973.332298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5973.342399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5974.888400] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5979.429866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5979.476286] Lustre: 401818:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5979.500628] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5979.503062] Lustre: Skipped 1 previous similar message [ 5979.542381] Lustre: lustre-MDT0001: new disk, initializing [ 5979.579353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5979.594617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5979.599640] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5981.117426] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5985.402431] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5987.789535] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5990.682612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5990.801616] Lustre: lustre-OST0000: new disk, initializing [ 5990.804151] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5990.807508] Lustre: 403790:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5990.842795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5993.246514] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 5997.347164] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5998.863218] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5999.616412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5999.625416] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5999.663765] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5999.969460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6001.584424] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6001.672595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6004.706589] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6004.706736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6004.713064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6004.715610] LustreError: Skipped 1 previous similar message [ 6004.718170] Lustre: Skipped 1 previous similar message [ 6008.547485] Lustre: server umount lustre-OST0000 complete [ 6010.335976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6010.339416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6010.344909] Lustre: Skipped 1 previous similar message [ 6010.347207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6014.944299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6014.944594] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6016.216745] Lustre: server umount lustre-MDT0000 complete [ 6017.616259] LustreError: 401810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788872048 with bad export cookie 14768037455532280121 [ 6017.619521] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6017.621258] LustreError: 401810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6017.742636] Lustre: server umount lustre-MDT0001 complete [ 6021.058806] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6021.809563] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 08:54:11 (1788872051) [ 6024.954945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6025.221728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6026.994034] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6030.304754] LustreError: 406253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6030.311862] LustreError: 406253:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6030.577918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6030.737775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6032.413404] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6035.412318] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6037.650563] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6040.473431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6040.626951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6042.950438] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6045.794631] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6046.562074] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6052.333771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6055.906396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6055.908207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6055.909153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6055.909158] Lustre: Skipped 1 previous similar message [ 6055.920049] Lustre: Skipped 1 previous similar message [ 6060.047811] Lustre: server umount lustre-OST0000 complete [ 6061.536489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6061.539597] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6061.546368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6061.549224] Lustre: Skipped 1 previous similar message [ 6067.604186] Lustre: server umount lustre-MDT0000 complete [ 6069.001430] LustreError: 406232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788872099 with bad export cookie 14768037455532280989 [ 6069.003442] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6069.006401] LustreError: 406232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6069.137662] Lustre: server umount lustre-MDT0001 complete [ 6073.096132] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 6074.187403] Key type lgssc unregistered [ 6074.328679] LNet: 409973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6074.332296] LNetError: 409973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6074.346200] LNet: Removed LNI 192.168.206.152@tcp [ 6074.720549] Key type .llcrypt unregistered [ 6074.722347] Key type ._llcrypt unregistered [ 6083.531680] Key type ._llcrypt registered [ 6083.532688] Key type .llcrypt registered [ 6083.570967] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 6083.671264] Key type .llcrypt unregistered [ 6083.672471] Key type ._llcrypt unregistered [ 6091.548752] Key type ._llcrypt registered [ 6091.549993] Key type .llcrypt registered [ 6091.589023] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 6091.919358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6091.931250] alg: No test for adler32 (adler32-zlib) [ 6092.812230] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 6092.928104] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 6094.519158] Key type lgssc registered [ 6094.963917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6097.793941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6097.800930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6098.941783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6100.391947] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6103.407335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6103.523262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6105.754773] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6108.640304] LustreError: 412148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6108.645490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6108.648056] LustreError: 412148:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6108.820991] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6111.427953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6111.627050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6113.348376] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6121.957187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6121.957403] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6121.965666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6121.973270] Lustre: Skipped 1 previous similar message [ 6127.075629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6127.082038] Lustre: Skipped 1 previous similar message [ 6127.458983] Lustre: server umount lustre-OST0000 complete [ 6132.193346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6132.193404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6132.199364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6132.210235] Lustre: Skipped 2 previous similar messages [ 6132.219554] Lustre: Skipped 1 previous similar message [ 6135.264365] Lustre: server umount lustre-MDT0000 complete [ 6137.088064] LustreError: 412133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788872167 with bad export cookie 81765280214076486 [ 6137.090689] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6137.096930] LustreError: 412133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6137.270293] Lustre: server umount lustre-MDT0001 complete [ 6142.488618] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 6143.943810] Key type lgssc unregistered [ 6144.127209] LNet: 415294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6144.133832] LNetError: 415294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6144.145116] LNet: Removed LNI 192.168.206.152@tcp [ 6144.526638] Key type .llcrypt unregistered [ 6144.528199] Key type ._llcrypt unregistered [ 6156.694318] Key type ._llcrypt registered [ 6156.695935] Key type .llcrypt registered [ 6156.751388] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 6157.126800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6157.143366] alg: No test for adler32 (adler32-zlib) [ 6158.055255] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 6158.179240] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 6159.783166] Key type lgssc registered [ 6160.384108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6164.127248] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6164.141619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6179.615391] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6184.737408] LustreError: 416659:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.152@tcp: failed processing log, type 4: rc = -110 [ 6211.359169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6213.546040] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6216.547330] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6219.578559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6219.822874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6219.853927] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6221.660766] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6224.864525] LustreError: 416685:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6224.872195] LustreError: 416685:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6225.246083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6225.418386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6226.934964] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6229.663111] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6231.872391] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6235.104108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6235.107396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6235.112676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6235.616369] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6235.619558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6235.624556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6236.643283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6236.645934] Lustre: Skipped 1 previous similar message [ 6240.542821] Lustre: server umount lustre-OST0000 complete [ 6244.833924] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6244.836907] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6244.843978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6246.880968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6248.024793] Lustre: server umount lustre-MDT0000 complete [ 6249.472656] LustreError: 419274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788872280 with bad export cookie 13411581217580594114 [ 6249.475186] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6249.477377] LustreError: 419274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6249.602197] Lustre: server umount lustre-MDT0001 complete [ 6253.736939] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 6255.034086] Key type lgssc unregistered [ 6255.201567] LNet: 420227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6255.207789] LNetError: 420227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6255.215714] LNet: Removed LNI 192.168.206.152@tcp [ 6255.602932] Key type .llcrypt unregistered [ 6255.604449] Key type ._llcrypt unregistered [ 6266.789288] Key type ._llcrypt registered [ 6266.791164] Key type .llcrypt registered [ 6266.837905] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 6267.246466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6267.299595] alg: No test for adler32 (adler32-zlib) [ 6268.165357] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 6268.259982] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 6269.847162] Key type lgssc registered [ 6270.300525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6273.450038] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6273.459958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6274.637901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6276.220885] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6279.745994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6279.905917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6281.549285] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6284.597546] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6286.907980] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6289.856506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6290.018078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6292.470367] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6295.613426] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6299.241145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6304.737406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6304.738839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6304.742822] Lustre: Skipped 2 previous similar messages [ 6304.749782] Lustre: Skipped 2 previous similar messages [ 6307.223565] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.52@tcp (stopping) [ 6309.247040] Lustre: server umount lustre-MDT0000 complete [ 6309.857735] LustreError: 421610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6309.865853] LustreError: 421610:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6310.339996] LustreError: 421595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.52@tcp arrived at 1788872341 with bad export cookie 3631463880262028283 [ 6310.349406] LustreError: 421595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6311.236636] LustreError: 422350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788872342 with bad export cookie 3631463880262027660 [ 6311.242040] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6311.242090] LustreError: 422350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6311.904167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6311.907178] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6311.912406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6314.975957] LustreError: 421616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6314.976646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6314.985558] LustreError: 421616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6317.023827] LustreError: 421615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6317.031636] LustreError: 421615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6317.349175] Lustre: server umount lustre-OST0000 complete [ 6320.378461] Lustre: server umount lustre-MDT0001 complete [ 6325.119938] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 6326.387429] Key type lgssc unregistered [ 6326.534364] LNet: 425293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6326.537705] LNetError: 425293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6326.545401] LNet: Removed LNI 192.168.206.152@tcp [ 6326.906402] Key type .llcrypt unregistered [ 6326.907805] Key type ._llcrypt unregistered [ 6337.290063] Key type ._llcrypt registered [ 6337.291146] Key type .llcrypt registered [ 6337.335082] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 6337.684547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6337.746393] alg: No test for adler32 (adler32-zlib) [ 6338.621529] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 6338.719702] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 6340.311153] Key type lgssc registered [ 6340.845883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6344.427205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6344.438136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6345.628344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6347.477220] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6351.335365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6351.554061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6353.633426] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6357.282075] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6359.927649] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6363.185747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6363.375669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6365.967527] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6369.446500] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6370.536119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6375.908433] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6375.910179] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6375.913984] Lustre: Skipped 1 previous similar message [ 6376.844381] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.52@tcp (stopping) [ 6376.850485] Lustre: Skipped 1 previous similar message [ 6378.786748] Lustre: server umount lustre-MDT0001 complete [ 6381.027355] LustreError: 427843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6381.039524] LustreError: 427843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6386.144711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6386.146055] LustreError: 426682:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6386.148445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6386.161344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6387.228407] Lustre: server umount lustre-OST0000 complete [ 6388.627152] Lustre: server umount lustre-MDT0000 complete [ 6393.622024] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 6394.803757] Key type lgssc unregistered [ 6394.942693] LNet: 430361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6394.946328] LNetError: 430361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6394.954401] LNet: Removed LNI 192.168.206.152@tcp [ 6395.301196] Key type .llcrypt unregistered [ 6395.302864] Key type ._llcrypt unregistered [ 6405.454826] Key type ._llcrypt registered [ 6405.457810] Key type .llcrypt registered [ 6405.507823] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 6412.302326] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 6412.666558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6412.677329] alg: No test for adler32 (adler32-zlib) [ 6413.546908] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 6413.654075] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 6415.247574] Key type lgssc registered [ 6415.709778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6430.531434] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 6436.805326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6436.814495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6437.918582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6437.931113] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6437.972904] Lustre: lustre-MDT0000: new disk, initializing [ 6438.005528] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6438.015085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6439.905658] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6447.052814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6447.164312] Lustre: lustre-OST0000: new disk, initializing [ 6447.166356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6447.169410] Lustre: Skipped 1 previous similar message [ 6447.172068] Lustre: 435373:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6447.204519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6449.411404] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6452.234396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6452.238405] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6452.251347] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6453.474852] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6460.063348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6460.124816] Lustre: 434355:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6460.163732] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6460.209433] Lustre: lustre-MDT0001: new disk, initializing [ 6460.266090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6460.290314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6460.296829] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6461.965342] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6465.902954] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6465.984132] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6467.522112] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6467.605718] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6469.540597] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.52@tcp (stopping) [ 6471.137694] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6471.137884] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6471.139052] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6471.143310] Lustre: Skipped 1 previous similar message [ 6474.633871] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.52@tcp (stopping) [ 6474.637726] Lustre: Skipped 1 previous similar message [ 6475.558303] Lustre: server umount lustre-MDT0001 complete [ 6481.377777] LustreError: 434362:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6481.383300] LustreError: 434362:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6486.497847] LustreError: 434389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6486.503303] LustreError: 434389:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6491.619102] LustreError: 437443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6491.619434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6491.633169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6494.506894] Lustre: server umount lustre-OST0000 complete [ 6502.281417] Lustre: server umount lustre-MDT0000 complete [ 6507.153724] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 6508.225443] Key type lgssc unregistered [ 6508.355356] LNet: 439113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6508.358217] LNetError: 439113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6508.367381] LNet: Removed LNI 192.168.206.152@tcp [ 6508.694271] Key type .llcrypt unregistered [ 6508.695869] Key type ._llcrypt unregistered [ 6523.598596] Key type ._llcrypt registered [ 6523.600326] Key type .llcrypt registered [ 6523.648365] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 6523.775248] Key type .llcrypt unregistered [ 6523.776758] Key type ._llcrypt unregistered [ 6531.210920] Key type ._llcrypt registered [ 6531.212363] Key type .llcrypt registered [ 6531.256607] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 6531.661556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6531.707849] alg: No test for adler32 (adler32-zlib) [ 6532.578513] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 6532.694904] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 6534.287149] Key type lgssc registered [ 6534.695704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6545.834660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6545.841518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6546.934855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6546.946653] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6546.983923] Lustre: lustre-MDT0000: new disk, initializing [ 6547.006403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6547.013376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6548.549687] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6553.105412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6553.150423] Lustre: 442467:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6553.177873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6553.181058] Lustre: Skipped 1 previous similar message [ 6553.220354] Lustre: lustre-MDT0001: new disk, initializing [ 6553.256584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6553.271847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6553.275552] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6554.871389] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6558.970304] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6561.022828] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6563.693029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6563.804345] Lustre: lustre-OST0000: new disk, initializing [ 6563.806745] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6563.812120] Lustre: 444437:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6563.850196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6566.023224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6566.028039] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6566.075127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6566.217789] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6570.533639] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6574.047969] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6574.051238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6574.056666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6576.609262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6576.609314] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6576.611422] Lustre: Skipped 1 previous similar message [ 6576.616612] Lustre: Skipped 1 previous similar message [ 6579.195346] Lustre: server umount lustre-MDT0000 complete [ 6580.764488] LustreError: 442459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788872611 with bad export cookie 5234248250225115806 [ 6580.767873] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6580.769332] LustreError: 442459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6580.880446] Lustre: server umount lustre-MDT0001 complete [ 6592.606983] Lustre: server umount lustre-OST0000 complete [ 6596.483015] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 09:03:45 (1788872625) [ 6599.380111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6599.589718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6601.203703] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6604.449611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6604.591101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6606.128146] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6608.797428] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6610.894553] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6626.402574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6626.477465] Lustre: Found index 0 for lustre-OST0000, updating log [ 6626.539227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6628.625503] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 6631.592438] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6645.988680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6650.335899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6650.339212] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6650.346247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6650.348866] Lustre: Skipped 1 previous similar message [ 6654.692976] Lustre: server umount lustre-OST0000 complete [ 6667.744350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6667.744880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6667.750530] Lustre: Skipped 2 previous similar messages [ 6667.753239] Lustre: Skipped 3 previous similar messages [ 6672.864309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6672.867087] Lustre: Skipped 1 previous similar message [ 6673.746977] Lustre: server umount lustre-MDT0000 complete [ 6675.202082] LustreError: 446443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788872706 with bad export cookie 5234248250225117199 [ 6675.206560] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6675.207454] LustreError: 446443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6675.338579] Lustre: server umount lustre-MDT0001 complete [ 6678.601977] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 09:05:07 (1788872707) [ 6693.471271] LNet: 1 local NIs in recovery (showing 1): 192.168.206.152@tcp [ 7263.144542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7263.363369] LustreError: 452887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7263.375030] LustreError: 452887:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7263.400140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7265.261226] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7268.833688] LustreError: 452888:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7269.162803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7269.352985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7271.445632] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7275.198377] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7278.223780] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7295.075691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7295.178839] Lustre: Found index 0 for lustre-OST0000, updating log [ 7295.312162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7298.023845] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7301.517384] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7302.626780] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7302.631284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7302.638239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7308.760082] Lustre: server umount lustre-OST0000 complete [ 7311.783613] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7311.785822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7311.793218] Lustre: Skipped 1 previous similar message [ 7313.889160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7313.894018] Lustre: Skipped 4 previous similar messages [ 7317.985334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7317.989044] Lustre: Skipped 6 previous similar messages [ 7321.183381] LNet: 1 local NIs in recovery (showing 1): 192.168.206.152@tcp [ 7325.151729] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7325.223682] Lustre: server umount lustre-MDT0000 complete [ 7326.178461] LustreError: 452887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7326.190639] LustreError: 452887:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7326.831534] LustreError: 452869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788873357 with bad export cookie 5234248250225118270 [ 7326.836246] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7326.836552] LustreError: 452869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7326.969922] Lustre: server umount lustre-MDT0001 complete [ 7346.926653] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 7351.368372] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 7368.025398] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 09:16:37 (1788873397) [ 7375.746542] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 7376.997257] Key type lgssc unregistered [ 7377.144495] LNet: 459530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7377.148566] LNetError: 459530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7427.364882] LNet: Removed LNI 192.168.206.152@tcp [ 7427.783812] Key type .llcrypt unregistered [ 7427.786536] Key type ._llcrypt unregistered [ 7435.733819] Key type ._llcrypt registered [ 7435.735812] Key type .llcrypt registered [ 7435.781169] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 7436.256324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7436.302794] alg: No test for adler32 (adler32-zlib) [ 7437.190489] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 7437.303398] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 7438.887170] Key type lgssc registered [ 7439.369900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7441.203057] LNet: Added LNI 192.168.206.152@tcp100 [8/256/0/180] [ 7441.560422] LNet: Added LNI 192.168.206.152@tcp101 [8/256/0/180] [ 7441.903833] LNet: Added LNI 192.168.206.152@tcp102 [8/256/0/180] [ 7442.288274] LNet: Added LNI 192.168.206.152@tcp103 [8/256/0/180] [ 7442.654991] LNet: Added LNI 192.168.206.152@tcp104 [8/256/0/180] [ 7443.008616] LNet: Added LNI 192.168.206.152@tcp105 [8/256/0/180] [ 7443.379304] LNet: Added LNI 192.168.206.152@tcp106 [8/256/0/180] [ 7443.722969] LNet: Added LNI 192.168.206.152@tcp107 [8/256/0/180] [ 7444.074529] LNet: Added LNI 192.168.206.152@tcp108 [8/256/0/180] [ 7444.410307] LNet: Added LNI 192.168.206.152@tcp109 [8/256/0/180] [ 7444.754429] LNet: Added LNI 192.168.206.152@tcp110 [8/256/0/180] [ 7445.121103] LNet: Added LNI 192.168.206.152@tcp111 [8/256/0/180] [ 7445.496646] LNet: Added LNI 192.168.206.152@tcp112 [8/256/0/180] [ 7445.826452] LNet: Added LNI 192.168.206.152@tcp113 [8/256/0/180] [ 7446.183437] LNet: Added LNI 192.168.206.152@tcp114 [8/256/0/180] [ 7446.549916] LNet: Added LNI 192.168.206.152@tcp115 [8/256/0/180] [ 7446.909580] LNet: Added LNI 192.168.206.152@tcp116 [8/256/0/180] [ 7447.259778] LNet: Added LNI 192.168.206.152@tcp117 [8/256/0/180] [ 7447.619878] LNet: Added LNI 192.168.206.152@tcp118 [8/256/0/180] [ 7447.964090] LNet: Added LNI 192.168.206.152@tcp119 [8/256/0/180] [ 7448.320751] LNet: Added LNI 192.168.206.152@tcp120 [8/256/0/180] [ 7448.669478] LNet: Added LNI 192.168.206.152@tcp121 [8/256/0/180] [ 7449.060934] LNet: Added LNI 192.168.206.152@tcp122 [8/256/0/180] [ 7449.445773] LNet: Added LNI 192.168.206.152@tcp123 [8/256/0/180] [ 7449.862375] LNet: Added LNI 192.168.206.152@tcp124 [8/256/0/180] [ 7450.247559] LNet: Added LNI 192.168.206.152@tcp125 [8/256/0/180] [ 7450.639523] LNet: Added LNI 192.168.206.152@tcp126 [8/256/0/180] [ 7451.039245] LNet: Added LNI 192.168.206.152@tcp127 [8/256/0/180] [ 7451.434227] LNet: Added LNI 192.168.206.152@tcp128 [8/256/0/180] [ 7451.807292] LNet: Added LNI 192.168.206.152@tcp129 [8/256/0/180] [ 7452.182815] LNet: Added LNI 192.168.206.152@tcp130 [8/256/0/180] [ 7452.538768] LNet: Added LNI 192.168.206.152@tcp131 [8/256/0/180] [ 7452.893310] LNet: Added LNI 192.168.206.152@tcp132 [8/256/0/180] [ 7453.271323] LNet: Added LNI 192.168.206.152@tcp133 [8/256/0/180] [ 7453.663171] LNet: Added LNI 192.168.206.152@tcp134 [8/256/0/180] [ 7454.028822] LNet: Added LNI 192.168.206.152@tcp135 [8/256/0/180] [ 7456.929291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7456.937636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7457.990276] Lustre: 462519:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.152@tcp: can't add NID 192.168.206.152@tcp130: rc = -75 [ 7458.015567] LNetError: 462539:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.152 [ 7458.019908] LNetError: Protocol error connecting to 192.168.206.152@tcp120 at host 192.168.206.152:988: is it running a compatible version of Lustre? [ 7458.024658] LNetError: Skipped 2 previous similar messages [ 7458.074769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7458.090962] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7458.131954] Lustre: lustre-MDT0000: new disk, initializing [ 7458.164129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7458.172521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7458.847287] LNet: 3 local NIs in recovery (showing 3): 192.168.206.152@tcp120, 192.168.206.152@tcp121, 192.168.206.152@tcp122 [ 7459.873487] LNetError: 459985:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.152 [ 7459.884789] LNetError: 459985:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [ 7459.892122] LNetError: Protocol error connecting to 192.168.206.152@tcp122 at host 192.168.206.152:988: is it running a compatible version of Lustre? [ 7459.898913] LNetError: Skipped 2 previous similar messages [ 7460.017314] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7461.921239] LNetError: 459982:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.152 [ 7461.925751] LNetError: Protocol error connecting to 192.168.206.152@tcp122 at host 192.168.206.152:988: is it running a compatible version of Lustre? [ 7464.888747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7464.953155] Lustre: 462537:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7464.986946] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7464.989679] Lustre: Skipped 1 previous similar message [ 7465.031297] Lustre: lustre-MDT0001: new disk, initializing [ 7465.043505] LNetError: 462539:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.152 [ 7465.048159] LNetError: Protocol error connecting to 192.168.206.152@tcp123 at host 192.168.206.152:988: is it running a compatible version of Lustre? [ 7465.048803] Lustre: 463342:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.152@tcp: can't add NID 192.168.206.152@tcp130: rc = -75 [ 7465.057253] Lustre: 463342:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7465.117474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7465.133972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7465.138550] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7466.490126] Lustre: 463548:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.152@tcp: can't add NID 192.168.206.152@tcp130: rc = -75 [ 7466.494411] Lustre: 463548:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7466.729450] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7469.090227] LNetError: 462541:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.152 [ 7469.094352] LNetError: 462541:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 33 previous similar messages [ 7469.098086] LNetError: Protocol error connecting to 192.168.206.152@tcp132 at host 192.168.206.152:988: is it running a compatible version of Lustre? [ 7469.103365] LNetError: Skipped 33 previous similar messages [ 7470.946382] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7473.170363] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7475.965883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7476.120748] Lustre: lustre-OST0000: new disk, initializing [ 7476.123457] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7476.125844] Lustre: 464516:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.152@tcp: can't add NID 192.168.206.152@tcp130: rc = -75 [ 7476.127194] Lustre: 464517:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7476.129334] Lustre: 464516:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [ 7476.211380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7478.647459] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7480.353454] LNetError: 459984:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.152 [ 7480.358871] LNetError: 459984:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [ 7480.362462] LNetError: Protocol error connecting to 192.168.206.152@tcp124 at host 192.168.206.152:988: is it running a compatible version of Lustre? [ 7480.366808] LNetError: Skipped 11 previous similar messages [ 7481.325738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7481.334708] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7481.359544] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7483.021514] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7486.434763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7486.435423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7486.436078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7486.447241] Lustre: Skipped 1 previous similar message [ 7488.224396] Lustre: server umount lustre-OST0000 complete [ 7491.041163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7491.044718] LustreError: Skipped 1 previous similar message [ 7491.046964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7491.056730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7491.060408] Lustre: Skipped 1 previous similar message [ 7495.852744] Lustre: server umount lustre-MDT0000 complete [ 7496.736441] LNetError: 459983:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.152 [ 7496.742086] LNetError: 459983:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [ 7496.747831] LNetError: Protocol error connecting to 192.168.206.152@tcp124 at host 192.168.206.152:988: is it running a compatible version of Lustre? [ 7496.752792] LNetError: Skipped 3 previous similar messages [ 7497.453337] LustreError: 462529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788873528 with bad export cookie 11767148929031674044 [ 7497.455839] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7497.459128] LustreError: 462529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7497.620713] Lustre: server umount lustre-MDT0001 complete [ 7502.256880] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 7503.440186] Key type lgssc unregistered [ 7503.623463] LNet: 466424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7503.627206] LNetError: 466424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7503.631069] LNet: Removed LNI 192.168.206.152@tcp [ 7504.671601] LNet: Removed LNI 192.168.206.152@tcp100 [ 7504.673393] LNetError: 466424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7504.676786] LNetError: 466424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7504.680779] LNet: Removed LNI 192.168.206.152@tcp101 [ 7505.695636] LNet: Removed LNI 192.168.206.152@tcp102 [ 7505.699218] LNetError: 466424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7505.703361] LNetError: 466424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7505.707147] LNet: Removed LNI 192.168.206.152@tcp103 [ 7506.719146] LNet: Removed LNI 192.168.206.152@tcp104 [ 7507.743211] LNet: Removed LNI 192.168.206.152@tcp105 [ 7507.745388] LNetError: 466424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7507.748633] LNetError: 466424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7507.752323] LNet: Removed LNI 192.168.206.152@tcp106 [ 7508.767110] LNet: Removed LNI 192.168.206.152@tcp107 [ 7509.791193] LNet: Removed LNI 192.168.206.152@tcp108 [ 7510.815137] LNet: Removed LNI 192.168.206.152@tcp109 [ 7511.839134] LNet: Removed LNI 192.168.206.152@tcp110 [ 7511.841234] LNetError: 466424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7511.844799] LNetError: 466424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7511.847702] LNet: Removed LNI 192.168.206.152@tcp111 [ 7512.863158] LNet: Removed LNI 192.168.206.152@tcp112 [ 7513.887300] LNet: Removed LNI 192.168.206.152@tcp113 [ 7514.911161] LNet: Removed LNI 192.168.206.152@tcp114 [ 7515.935153] LNet: Removed LNI 192.168.206.152@tcp115 [ 7516.959113] LNet: Removed LNI 192.168.206.152@tcp116 [ 7517.983488] LNet: Removed LNI 192.168.206.152@tcp117 [ 7519.007117] LNet: Removed LNI 192.168.206.152@tcp118 [ 7520.031276] LNet: Removed LNI 192.168.206.152@tcp119 [ 7520.034583] LNetError: 466424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7520.040200] LNetError: 466424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 7520.043095] LNet: Removed LNI 192.168.206.152@tcp120 [ 7521.055145] LNet: Removed LNI 192.168.206.152@tcp121 [ 7521.057855] LNet: Removed LNI 192.168.206.152@tcp122 [ 7522.079195] LNet: Removed LNI 192.168.206.152@tcp123 [ 7522.080689] LNet: Removed LNI 192.168.206.152@tcp124 [ 7523.103538] LNet: Removed LNI 192.168.206.152@tcp125 [ 7524.127129] LNet: Removed LNI 192.168.206.152@tcp126 [ 7525.151520] LNet: Removed LNI 192.168.206.152@tcp127 [ 7526.175146] LNet: Removed LNI 192.168.206.152@tcp128 [ 7527.199140] LNet: Removed LNI 192.168.206.152@tcp129 [ 7528.223362] LNet: Removed LNI 192.168.206.152@tcp130 [ 7529.247130] LNet: Removed LNI 192.168.206.152@tcp131 [ 7529.249173] LNet: Removed LNI 192.168.206.152@tcp132 [ 7530.271148] LNet: Removed LNI 192.168.206.152@tcp133 [ 7531.295190] LNet: Removed LNI 192.168.206.152@tcp134 [ 7532.327074] LNet: Removed LNI 192.168.206.152@tcp135 [ 7532.676218] Key type .llcrypt unregistered [ 7532.677792] Key type ._llcrypt unregistered [ 7542.302931] Key type ._llcrypt registered [ 7542.304395] Key type .llcrypt registered [ 7542.348252] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 7548.931310] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 7549.278912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7549.290093] alg: No test for adler32 (adler32-zlib) [ 7550.171251] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 7550.278472] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 7551.879517] Key type lgssc registered [ 7552.337540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7565.149937] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 09:19:54 (1788873594) [ 7572.266339] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 7573.568044] Key type lgssc unregistered [ 7573.710636] LNet: 470478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7573.715631] LNetError: 470478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7573.726600] LNet: Removed LNI 192.168.206.152@tcp [ 7574.086505] Key type .llcrypt unregistered [ 7574.088103] Key type ._llcrypt unregistered [ 7581.985842] Key type ._llcrypt registered [ 7581.987410] Key type .llcrypt registered [ 7582.042194] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 7582.506418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7582.514294] alg: No test for adler32 (adler32-zlib) [ 7583.388529] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 7583.493219] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 7585.103122] Key type lgssc registered [ 7585.564729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7589.569707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7589.582027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7590.691331] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7590.709962] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7590.750991] Lustre: lustre-MDT0000: new disk, initializing [ 7590.788403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7590.799587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7592.667407] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7597.988912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7598.052532] Lustre: 471679:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7598.097806] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7598.105017] Lustre: Skipped 1 previous similar message [ 7598.151249] Lustre: lustre-MDT0001: new disk, initializing [ 7598.213562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7598.232891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7598.240472] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7600.365639] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7605.324722] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7607.880851] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7610.898157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7611.016246] Lustre: lustre-OST0000: new disk, initializing [ 7611.018526] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7611.022775] Lustre: 473647:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7611.060269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7612.806233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7612.810846] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7612.857910] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7613.643869] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7618.053776] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7619.618466] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing lnet_if_list [ 7621.786572] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7621.802284] LNet: Added LNI 192.168.206.152@tcp10 [8/256/0/180] [ 7624.199935] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7624.220678] LNet: Added LNI 192.168.206.152@tcp11 [8/256/0/180] [ 7626.163326] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7626.179660] LNet: Added LNI 192.168.206.152@tcp12 [8/256/0/180] [ 7627.943210] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7627.958022] LNet: Added LNI 192.168.206.152@tcp13 [8/256/0/180] [ 7629.792117] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7629.809640] LNet: Added LNI 192.168.206.152@tcp14 [8/256/0/180] [ 7631.782949] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7631.799924] LNet: Added LNI 192.168.206.152@tcp15 [8/256/0/180] [ 7633.731978] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7633.749848] LNet: Added LNI 192.168.206.152@tcp16 [8/256/0/180] [ 7635.895257] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7635.912452] LNet: Added LNI 192.168.206.152@tcp17 [8/256/0/180] [ 7638.085888] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7638.100519] LNet: Added LNI 192.168.206.152@tcp18 [8/256/0/180] [ 7640.225890] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7640.242035] LNet: Added LNI 192.168.206.152@tcp19 [8/256/0/180] [ 7642.410323] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7642.423912] LNet: Added LNI 192.168.206.152@tcp20 [8/256/0/180] [ 7644.420165] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7644.434776] LNet: Added LNI 192.168.206.152@tcp21 [8/256/0/180] [ 7646.535201] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7646.550149] LNet: Added LNI 192.168.206.152@tcp22 [8/256/0/180] [ 7648.616748] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7648.632478] LNet: Added LNI 192.168.206.152@tcp23 [8/256/0/180] [ 7650.894774] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7650.909631] LNet: Added LNI 192.168.206.152@tcp24 [8/256/0/180] [ 7653.208922] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7653.223226] LNet: Added LNI 192.168.206.152@tcp25 [8/256/0/180] [ 7655.639531] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7655.656044] LNet: Added LNI 192.168.206.152@tcp26 [8/256/0/180] [ 7658.114122] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7658.131577] LNet: Added LNI 192.168.206.152@tcp27 [8/256/0/180] [ 7660.638516] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7660.655021] LNet: Added LNI 192.168.206.152@tcp28 [8/256/0/180] [ 7663.075912] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7663.091910] LNet: Added LNI 192.168.206.152@tcp29 [8/256/0/180] [ 7665.553252] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7665.568836] LNet: Added LNI 192.168.206.152@tcp30 [8/256/0/180] [ 7668.129370] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7668.142629] LNet: Added LNI 192.168.206.152@tcp31 [8/256/0/180] [ 7670.641500] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7670.656136] LNet: Added LNI 192.168.206.152@tcp32 [8/256/0/180] [ 7673.220938] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7673.237961] LNet: Added LNI 192.168.206.152@tcp33 [8/256/0/180] [ 7675.932365] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7675.947742] LNet: Added LNI 192.168.206.152@tcp34 [8/256/0/180] [ 7678.610405] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7678.624934] LNet: Added LNI 192.168.206.152@tcp35 [8/256/0/180] [ 7681.473141] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7681.489724] LNet: Added LNI 192.168.206.152@tcp36 [8/256/0/180] [ 7684.692506] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7684.706721] LNet: Added LNI 192.168.206.152@tcp37 [8/256/0/180] [ 7688.091688] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7688.107463] LNet: Added LNI 192.168.206.152@tcp38 [8/256/0/180] [ 7691.182251] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7691.197490] LNet: Added LNI 192.168.206.152@tcp39 [8/256/0/180] [ 7694.178829] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7694.193915] LNet: Added LNI 192.168.206.152@tcp40 [8/256/0/180] [ 7696.990517] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7697.006364] LNet: Added LNI 192.168.206.152@tcp41 [8/256/0/180] [ 7699.849025] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7699.862573] LNet: Added LNI 192.168.206.152@tcp42 [8/256/0/180] [ 7702.827197] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7702.839809] LNet: Added LNI 192.168.206.152@tcp43 [8/256/0/180] [ 7705.791586] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7705.805313] LNet: Added LNI 192.168.206.152@tcp44 [8/256/0/180] [ 7708.933509] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7708.949505] LNet: Added LNI 192.168.206.152@tcp45 [8/256/0/180] [ 7711.877691] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7711.892568] LNet: Added LNI 192.168.206.152@tcp46 [8/256/0/180] [ 7714.824056] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7714.839353] LNet: Added LNI 192.168.206.152@tcp47 [8/256/0/180] [ 7717.935562] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7717.951186] LNet: Added LNI 192.168.206.152@tcp48 [8/256/0/180] [ 7721.508338] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7721.523822] LNet: Added LNI 192.168.206.152@tcp49 [8/256/0/180] [ 7725.286500] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7725.303714] LNet: Added LNI 192.168.206.152@tcp50 [8/256/0/180] [ 7735.778084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7735.779896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7735.783543] Lustre: Skipped 1 previous similar message [ 7740.909794] Lustre: server umount lustre-MDT0000 complete [ 7742.407222] LustreError: 473646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788873773 with bad export cookie 16274859658724699160 [ 7742.410612] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7742.412163] LustreError: 473646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7742.533343] Lustre: server umount lustre-MDT0001 complete [ 7750.650473] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 7754.284909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7761.375216] Lustre: 470951:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788873776/real 1788873776] req@00000000d7bd7dd1 x1875769938766080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788873792 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7761.390116] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7761.395674] Lustre: Skipped 1 previous similar message [ 7762.399173] Lustre: 470952:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788873777/real 1788873777] req@00000000f2c52105 x1875769938766208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788873793 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7766.495165] Lustre: 470949:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788873781/real 1788873781] req@00000000a898ebca x1875769938766336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788873797 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7768.544090] LustreError: 470948:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000804c9151 x1875769938767104/t0(0) o250->MGC192.168.206.152@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7768.666177] LustreError: 473639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7768.676105] LustreError: 473639:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7768.682981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7769.695161] Lustre: 470949:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788873784/real 1788873784] req@000000006b3bccff x1875769938766592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788873800 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7770.298314] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7770.720520] LustreError: 473639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7771.746225] LustreError: 497831:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7771.753506] LustreError: 497831:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7773.815627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7774.018972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7775.072121] Lustre: 470952:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788873789/real 1788873789] req@00000000a75f10a1 x1875769938766720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788873805 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7775.679266] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7778.366296] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7779.304074] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7780.529725] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7789.544503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7789.544660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7789.548386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7789.555237] Lustre: Skipped 1 previous similar message [ 7789.561589] Lustre: Skipped 3 previous similar messages [ 7794.662298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7794.665037] Lustre: Skipped 1 previous similar message [ 7795.164116] Lustre: server umount lustre-OST0000 complete [ 7799.776861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7799.776935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7799.780810] LustreError: Skipped 1 previous similar message [ 7799.788620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7799.792141] Lustre: Skipped 1 previous similar message [ 7804.896888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7804.900130] Lustre: Skipped 1 previous similar message [ 7811.039120] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7811.105791] Lustre: server umount lustre-MDT0000 complete [ 7812.592414] LustreError: 471670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788873843 with bad export cookie 16274859658724701302 [ 7812.595871] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7812.596978] LustreError: 471670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7812.717360] Lustre: server umount lustre-MDT0001 complete [ 7817.045834] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 7818.346098] Key type lgssc unregistered [ 7818.541056] LNet: 500611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7818.545401] LNetError: 500611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7818.549844] LNet: Removed LNI 192.168.206.152@tcp [ 7818.551819] LNet: Removed LNI 192.168.206.152@tcp10 [ 7818.553506] LNet: Removed LNI 192.168.206.152@tcp11 [ 7818.555624] LNet: Removed LNI 192.168.206.152@tcp12 [ 7818.557646] LNet: Removed LNI 192.168.206.152@tcp13 [ 7818.559468] LNet: Removed LNI 192.168.206.152@tcp14 [ 7818.561275] LNet: Removed LNI 192.168.206.152@tcp15 [ 7818.562715] LNet: Removed LNI 192.168.206.152@tcp16 [ 7818.564327] LNet: Removed LNI 192.168.206.152@tcp17 [ 7818.565704] LNet: Removed LNI 192.168.206.152@tcp18 [ 7818.567447] LNet: Removed LNI 192.168.206.152@tcp19 [ 7818.568693] LNet: Removed LNI 192.168.206.152@tcp20 [ 7818.570195] LNet: Removed LNI 192.168.206.152@tcp21 [ 7818.571852] LNet: Removed LNI 192.168.206.152@tcp22 [ 7818.573780] LNet: Removed LNI 192.168.206.152@tcp23 [ 7818.575813] LNet: Removed LNI 192.168.206.152@tcp24 [ 7818.577587] LNet: Removed LNI 192.168.206.152@tcp25 [ 7818.579500] LNet: Removed LNI 192.168.206.152@tcp26 [ 7818.581737] LNet: Removed LNI 192.168.206.152@tcp27 [ 7818.583666] LNet: Removed LNI 192.168.206.152@tcp28 [ 7818.585819] LNet: Removed LNI 192.168.206.152@tcp29 [ 7818.588208] LNet: Removed LNI 192.168.206.152@tcp30 [ 7818.590834] LNet: Removed LNI 192.168.206.152@tcp31 [ 7818.592988] LNet: Removed LNI 192.168.206.152@tcp32 [ 7818.595545] LNet: Removed LNI 192.168.206.152@tcp33 [ 7818.598280] LNet: Removed LNI 192.168.206.152@tcp34 [ 7818.600075] LNet: Removed LNI 192.168.206.152@tcp35 [ 7818.601643] LNet: Removed LNI 192.168.206.152@tcp36 [ 7818.603202] LNet: Removed LNI 192.168.206.152@tcp37 [ 7818.605438] LNet: Removed LNI 192.168.206.152@tcp38 [ 7818.608161] LNet: Removed LNI 192.168.206.152@tcp39 [ 7818.610190] LNet: Removed LNI 192.168.206.152@tcp40 [ 7818.611835] LNet: Removed LNI 192.168.206.152@tcp41 [ 7818.614205] LNet: Removed LNI 192.168.206.152@tcp42 [ 7818.615482] LNet: Removed LNI 192.168.206.152@tcp43 [ 7818.617223] LNet: Removed LNI 192.168.206.152@tcp44 [ 7818.619626] LNet: Removed LNI 192.168.206.152@tcp45 [ 7818.621911] LNet: Removed LNI 192.168.206.152@tcp46 [ 7818.623832] LNet: Removed LNI 192.168.206.152@tcp47 [ 7818.626152] LNet: Removed LNI 192.168.206.152@tcp48 [ 7818.628426] LNet: Removed LNI 192.168.206.152@tcp49 [ 7818.638332] LNet: Removed LNI 192.168.206.152@tcp50 [ 7819.014656] Key type .llcrypt unregistered [ 7819.016068] Key type ._llcrypt unregistered [ 7827.644298] Key type ._llcrypt registered [ 7827.646586] Key type .llcrypt registered [ 7827.713049] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 7835.078265] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 7835.667901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7835.678256] alg: No test for adler32 (adler32-zlib) [ 7836.575971] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 7836.687990] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 7838.295224] Key type lgssc registered [ 7838.766552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7852.833351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7852.843335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7853.974172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7853.993396] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7854.035731] Lustre: lustre-MDT0000: new disk, initializing [ 7854.069587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7854.077227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7855.866427] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7860.938670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7860.988812] Lustre: 503777:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7861.017249] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7861.020027] Lustre: Skipped 1 previous similar message [ 7861.064885] Lustre: lustre-MDT0001: new disk, initializing [ 7861.119040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7861.138768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7861.142676] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7862.896201] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7868.027612] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7870.674479] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7874.971323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7875.104857] Lustre: *** cfs_fail_loc=90f, val=0*** [ 7875.196298] Lustre: lustre-OST0000: new disk, initializing [ 7875.199613] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7875.210721] Lustre: 505793:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7875.272075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7878.429159] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7883.333768] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7884.304295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7884.307851] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7884.365897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7889.379886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7889.380470] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7889.390443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7889.393048] LustreError: Skipped 1 previous similar message [ 7889.398295] Lustre: Skipped 1 previous similar message [ 7892.839817] Lustre: server umount lustre-OST0000 complete [ 7894.496549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7894.501271] Lustre: Skipped 1 previous similar message [ 7894.504236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7894.509513] Lustre: Skipped 1 previous similar message [ 7899.616613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7899.619939] Lustre: Skipped 1 previous similar message [ 7900.569366] Lustre: server umount lustre-MDT0000 complete [ 7902.045234] LustreError: 505792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788873932 with bad export cookie 6958379562692245778 [ 7902.048407] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7902.052781] LustreError: 505792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7902.192858] Lustre: server umount lustre-MDT0001 complete [ 7906.805214] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 7908.081807] Key type lgssc unregistered [ 7908.232963] LNet: 507694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7908.237944] LNetError: 507694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7908.250178] LNet: Removed LNI 192.168.206.152@tcp [ 7908.635334] Key type .llcrypt unregistered [ 7908.636959] Key type ._llcrypt unregistered [ 7917.805706] Key type ._llcrypt registered [ 7917.807643] Key type .llcrypt registered [ 7917.850658] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 7918.304034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7918.316880] alg: No test for adler32 (adler32-zlib) [ 7919.194094] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 7919.305324] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 7920.895169] Key type lgssc registered [ 7921.412466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7924.716962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7924.725586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7925.904744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7927.499483] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7930.970482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7931.155413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7932.836387] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7935.585508] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7937.551886] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7940.215392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7940.359517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7942.795241] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7945.788241] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7955.474781] loop0: detected capacity change from 0 to 204800000 [ 7955.540613] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7955.545842] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7955.551568] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7957.899815] loop0: detected capacity change from 0 to 204800000 [ 7957.964437] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7957.968831] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7957.973931] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7959.932405] loop0: detected capacity change from 0 to 204800000 [ 7962.170777] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7962.245949] Lustre: fs15246-OST0000: new disk, initializing [ 7962.248819] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7962.253385] Lustre: 512529:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7962.285591] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7964.592210] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7969.033062] loop1: detected capacity change from 0 to 204800000 [ 7971.342535] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7971.389406] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7971.418444] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7971.458363] Lustre: fs15246-MDT0000: new disk, initializing [ 7971.525653] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7971.539425] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7971.582671] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7972.641948] Lustre: 513649:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000ecc6a4fb x1875770291026816/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788874019 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7973.350218] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 7977.699712] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7977.704497] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7977.712805] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7987.689480] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7987.695100] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7988.853562] Lustre: server umount fs15246-MDT0000 complete [ 7992.801817] LustreError: 508935:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7992.811024] LustreError: 508935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7993.175854] Lustre: server umount fs15246-OST0000 complete [ 7997.922867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7997.924061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7997.932384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7997.935567] Lustre: Skipped 1 previous similar message [ 8003.042632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8003.046840] Lustre: Skipped 1 previous similar message [ 8003.229548] Lustre: server umount lustre-OST0000 complete [ 8008.161359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8008.161491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8008.169589] Lustre: Skipped 1 previous similar message [ 8008.172303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8008.175290] Lustre: Skipped 1 previous similar message [ 8010.913299] Lustre: server umount lustre-MDT0000 complete [ 8012.653148] LustreError: 514786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788874043 with bad export cookie 8096480770960696135 [ 8012.655661] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8012.661505] LustreError: 514786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8012.798438] Lustre: server umount lustre-MDT0001 complete [ 8017.429249] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 8018.646918] Key type lgssc unregistered [ 8018.794419] LNet: 516742:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8018.798917] LNetError: 516742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8018.812513] LNet: Removed LNI 192.168.206.152@tcp [ 8019.191076] Key type .llcrypt unregistered [ 8019.192702] Key type ._llcrypt unregistered [ 8027.159904] Key type ._llcrypt registered [ 8027.161415] Key type .llcrypt registered [ 8027.204438] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 8027.633561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8027.659665] alg: No test for adler32 (adler32-zlib) [ 8028.529686] Lustre: Lustre: Build Version: 2.17.57_104_g1a15c94 [ 8028.626456] LNet: Added LNI 192.168.206.152@tcp [8/256/0/180] [ 8030.215323] Key type lgssc registered [ 8030.671039] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8046.065843] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 8049.934083] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 8064.531485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8064.550864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8065.674725] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8065.700808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8065.743538] Lustre: lustre-MDT0000: new disk, initializing [ 8065.788878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8065.797756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8067.545686] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 8072.644094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8072.706993] Lustre: 521173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8072.739964] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8072.742767] Lustre: Skipped 1 previous similar message [ 8072.787022] Lustre: lustre-MDT0001: new disk, initializing [ 8072.831950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8072.849401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8072.854731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8074.758891] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 8079.253923] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8081.491601] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8084.431204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8084.576302] Lustre: lustre-OST0000: new disk, initializing [ 8084.578735] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8084.583088] Lustre: 523139:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8084.628982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8085.192682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8085.196689] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8085.234855] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8087.191323] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_default_debug -1 all [ 8091.770592] Lustre: DEBUG MARKER: oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8093.669437] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8093.758038] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8095.530586] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8095.631716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8100.838742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8100.839689] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8100.850875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8100.851072] LustreError: Skipped 1 previous similar message [ 8100.857945] Lustre: Skipped 1 previous similar message [ 8102.555900] Lustre: server umount lustre-OST0000 complete [ 8105.953035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8105.954268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8105.961503] Lustre: Skipped 2 previous similar messages [ 8110.314868] Lustre: server umount lustre-MDT0000 complete [ 8111.855211] LustreError: 521968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788874142 with bad export cookie 2094126372061196435 [ 8111.857824] LustreError: MGC192.168.206.152@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8111.862083] LustreError: 521968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8112.005612] Lustre: server umount lustre-MDT0001 complete [ 8121.802483] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing set_hostid [ 8126.100735] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing load_modules_local [ 8139.273130] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8012 sec ========== 09:29:28 (1788874168) [ 8139.944244] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 09:29:29 (1788874169) === [ 8141.341196] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 09:29:30 (1788874170) === [ 8156.396475] Lustre: DEBUG MARKER: oleg652-server.virtnet: executing unload_modules_local [ 8157.653630] Key type lgssc unregistered [ 8157.808445] LNet: 529770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8157.811267] LNetError: 529770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8157.818441] LNet: Removed LNI 192.168.206.152@tcp [ 8158.180739] Key type .llcrypt unregistered [ 8158.182768] Key type ._llcrypt unregistered