[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 542331070 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002358] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.007499] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009000] pid_max: default: 32768 minimum: 301 [ 0.009133] LSM: Security Framework initializing [ 0.010048] Yama: becoming mindful. [ 0.011029] SELinux: Initializing. [ 0.012069] *** VALIDATE selinux *** [ 0.021246] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025790] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029104] *** VALIDATE tmpfs *** [ 0.030454] *** VALIDATE proc *** [ 0.031220] *** VALIDATE cgroup *** [ 0.032007] *** VALIDATE cgroup2 *** [ 0.033238] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035027] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037024] Spectre V2 : User space: Vulnerable [ 0.038006] Speculative Store Bypass: Vulnerable [ 0.041179] debug: unmapping init [mem 0xffffffffa8059000-0xffffffffa8060fff] [ 0.043251] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044628] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045018] ... version: 2 [ 0.046008] ... bit width: 48 [ 0.047007] ... generic registers: 4 [ 0.048009] ... value mask: 0000ffffffffffff [ 0.049024] ... max period: 00007fffffffffff [ 0.050014] ... fixed-purpose events: 3 [ 0.051009] ... event mask: 000000070000000f [ 0.052266] rcu: Hierarchical SRCU implementation. [ 0.054415] smp: Bringing up secondary CPUs ... [ 0.055548] x86: Booting SMP configuration: [ 0.056015] .... node #0, CPUs: #1 #2 #3 [ 0.060298] smp: Brought up 1 node, 4 CPUs [ 0.062016] smpboot: Max logical packages: 1 [ 0.063008] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.133200] node 0 deferred pages initialised in 68ms [ 0.137243] devtmpfs: initialized [ 0.139404] x86/mm: Memory block size: 128MB [ 0.141830] gcov: version magic: 0x41383552 [ 0.144307] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.146061] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.149333] pinctrl core: initialized pinctrl subsystem [ 0.151123] [ 0.151534] ************************************************************* [ 0.154006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156006] ** ** [ 0.158007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160007] ** ** [ 0.162007] ** This means that this kernel is built to expose internal ** [ 0.164007] ** IOMMU data structures, which may compromise security on ** [ 0.166007] ** your system. ** [ 0.169006] ** ** [ 0.171006] ** If you see this message and you are not debugging the ** [ 0.173017] ** kernel, report this immediately to your vendor! ** [ 0.175007] ** ** [ 0.177008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.179006] ************************************************************* [ 0.191272] NET: Registered protocol family 16 [ 0.193536] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.196046] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.198043] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.201046] cpuidle: using governor menu [ 0.202855] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.205687] PCI: Using configuration type 1 for base access [ 0.207178] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.217387] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.219012] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.222377] cryptd: max_cpu_qlen set to 1000 [ 0.224000] ACPI: Added _OSI(Module Device) [ 0.224000] ACPI: Added _OSI(Processor Device) [ 0.224000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.224000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.230984] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.237650] ACPI: Interpreter enabled [ 0.239048] ACPI: PM: (supports S0 S3 S4 S5) [ 0.241009] ACPI: Using IOAPIC for interrupt routing [ 0.242156] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.246000] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.265963] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.268061] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.272008] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.276131] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.281467] acpiphp: Slot [2] registered [ 0.284125] acpiphp: Slot [5] registered [ 0.286248] acpiphp: Slot [6] registered [ 0.288388] acpiphp: Slot [7] registered [ 0.292173] acpiphp: Slot [8] registered [ 0.293114] acpiphp: Slot [9] registered [ 0.294173] acpiphp: Slot [10] registered [ 0.296114] acpiphp: Slot [3] registered [ 0.297096] acpiphp: Slot [4] registered [ 0.299112] acpiphp: Slot [11] registered [ 0.300102] acpiphp: Slot [12] registered [ 0.302105] acpiphp: Slot [13] registered [ 0.304113] acpiphp: Slot [14] registered [ 0.306129] acpiphp: Slot [15] registered [ 0.307097] acpiphp: Slot [16] registered [ 0.309171] acpiphp: Slot [17] registered [ 0.310119] acpiphp: Slot [18] registered [ 0.312109] acpiphp: Slot [19] registered [ 0.313104] acpiphp: Slot [20] registered [ 0.314104] acpiphp: Slot [21] registered [ 0.315000] acpiphp: Slot [22] registered [ 0.315000] acpiphp: Slot [23] registered [ 0.315085] acpiphp: Slot [24] registered [ 0.316000] acpiphp: Slot [25] registered [ 0.318057] acpiphp: Slot [26] registered [ 0.319071] acpiphp: Slot [27] registered [ 0.320037] acpiphp: Slot [28] registered [ 0.321093] acpiphp: Slot [29] registered [ 0.325218] acpiphp: Slot [30] registered [ 0.327329] acpiphp: Slot [31] registered [ 0.328000] PCI host bridge to bus 0000:00 [ 0.328000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.328000] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.328000] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.328000] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.329019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.330000] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.336167] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.342069] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.345628] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.358947] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.365588] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.368024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.370014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.372022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.375556] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.377851] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.380086] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.382871] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.392015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.399016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.407013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.414026] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.419011] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.425014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.448016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.464573] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.483017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.496015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.521019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.536619] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.546014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.551012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.565017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.579317] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.589012] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.599014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.616014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.627511] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.645015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.663015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.708015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.729075] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.739014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.741000] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.763015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.770000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.770000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.770000] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.770000] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.770000] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.770220] iommu: Default domain type: Passthrough [ 0.771551] SCSI subsystem initialized [ 0.772000] ACPI: bus type USB registered [ 0.772000] usbcore: registered new interface driver usbfs [ 0.772000] usbcore: registered new interface driver hub [ 0.778060] usbcore: registered new device driver usb [ 0.780135] pps_core: LinuxPPS API ver. 1 registered [ 0.782006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.785045] PTP clock support registered [ 0.787257] EDAC MC: Ver: 3.0.0 [ 0.788679] PCI: Using ACPI for IRQ routing [ 0.790901] NetLabel: Initializing [ 0.792008] NetLabel: domain hash size = 128 [ 0.793006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.795063] NetLabel: unlabeled traffic allowed by default [ 0.797242] vgaarb: loaded [ 0.799103] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.801006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.808465] clocksource: Switched to clocksource kvm-clock [ 0.923588] VFS: Disk quotas dquot_6.6.0 [ 0.925365] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.927757] *** VALIDATE ramfs *** [ 0.928836] *** VALIDATE hugetlbfs *** [ 0.930539] pnp: PnP ACPI init [ 0.933324] pnp: PnP ACPI: found 6 devices [ 0.948291] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.951224] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.953179] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.955134] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.957283] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.959452] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.962110] NET: Registered protocol family 2 [ 0.964383] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.968960] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.971871] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.976409] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.979201] TCP: Hash tables configured (established 65536 bind 65536) [ 0.982029] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.985204] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.987823] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.991244] NET: Registered protocol family 1 [ 0.993783] RPC: Registered named UNIX socket transport module. [ 0.995984] RPC: Registered udp transport module. [ 1.000088] RPC: Registered tcp transport module. [ 1.001807] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.004974] NET: Registered protocol family 44 [ 1.006588] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.009071] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.011429] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.014079] PCI: CLS 0 bytes, default 64 [ 1.016609] Unpacking initramfs... [ 2.682469] debug: unmapping init [mem 0xffff8ba0bcc54000-0xffff8ba0bffbffff] [ 2.686832] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.689286] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.692382] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 3.206676] Initialise system trusted keyrings [ 3.207936] Key type blacklist registered [ 3.211217] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.219845] zbud: loaded [ 3.222608] *** VALIDATE nfs *** [ 3.223902] *** VALIDATE nfs4 *** [ 3.225563] pstore: using deflate compression [ 3.229441] Platform Keyring initialized [ 3.352652] NET: Registered protocol family 38 [ 3.354966] Key type asymmetric registered [ 3.356729] Asymmetric key parser 'x509' registered [ 3.359906] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.366477] io scheduler mq-deadline registered [ 3.369540] io scheduler kyber registered [ 3.371564] io scheduler bfq registered [ 3.374685] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.378497] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.381454] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.384176] ACPI: Power Button [PWRF] [ 3.391212] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.398306] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.415307] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.421715] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.432706] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.459056] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.485285] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.491341] Non-volatile memory driver v1.3 [ 3.495453] Linux agpgart interface v0.103 [ 3.532760] virtio_blk virtio1: [vda] 146504 512-byte logical blocks (75.0 MB/71.5 MiB) [ 3.538387] vda: detected capacity change from 0 to 75010048 [ 3.569461] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.573049] vdb: detected capacity change from 0 to 1073741824 [ 3.593713] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.595915] vdc: detected capacity change from 0 to 2621440000 [ 3.618196] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.622405] vdd: detected capacity change from 0 to 2621440000 [ 3.639664] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.643273] vde: detected capacity change from 0 to 4294967296 [ 3.665486] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.669492] vdf: detected capacity change from 0 to 4294967296 [ 3.679710] libphy: Fixed MDIO Bus: probed [ 3.688522] usbcore: registered new interface driver usbserial_generic [ 3.691812] usbserial: USB Serial support registered for generic [ 3.695227] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.700687] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.702834] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.705676] mousedev: PS/2 mouse device common for all mice [ 3.709241] rtc_cmos 00:05: RTC can wake from S4 [ 3.711262] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.716528] rtc_cmos 00:05: registered as rtc0 [ 3.720247] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.723675] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.725062] intel_pstate: CPU model not supported [ 3.734774] hid: raw HID events driver (C) Jiri Kosina [ 3.736688] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.738875] usbcore: registered new interface driver usbhid [ 3.743428] usbhid: USB HID core driver [ 3.744935] drop_monitor: Initializing network drop monitor service [ 3.746636] Initializing XFRM netlink socket [ 3.748128] NET: Registered protocol family 10 [ 3.750486] Segment Routing with IPv6 [ 3.751515] NET: Registered protocol family 17 [ 3.752901] mpls_gso: MPLS GSO support [ 3.757070] RAS: Correctable Errors collector initialized. [ 3.758709] AVX version of gcm_enc/dec engaged. [ 3.760280] AES CTR mode by8 optimization enabled [ 3.826411] sched_clock: Marking stable (3826337370, 0)->(4790384439, -964047069) [ 3.829899] registered taskstats version 1 [ 3.830970] Loading compiled-in X.509 certificates [ 3.832522] zswap: loaded using pool lzo/zbud [ 3.854555] Key type big_key registered [ 3.867585] Key type encrypted registered [ 3.868815] ima: No TPM chip found, activating TPM-bypass! [ 3.871136] ima: Allocated hash algorithm: sha1 [ 3.872495] ima: No architecture policies found [ 3.874199] evm: Initialising EVM extended attributes: [ 3.875761] evm: security.selinux [ 3.876645] evm: security.ima [ 3.877782] evm: security.capability [ 3.879363] evm: HMAC attrs: 0x1 [ 3.881681] rtc_cmos 00:05: setting system clock to 2026-06-28 07:08:41 UTC (1782630521) [ 3.886108] debug: unmapping init [mem 0xffffffffa9003000-0xffffffffa91fffff] [ 3.888211] debug: unmapping init [mem 0xffffffffa7d82000-0xffffffffa8058fff] [ 3.897726] Write protecting the kernel read-only data: 28672k [ 3.901295] debug: unmapping init [mem 0xffffffffa6403000-0xffffffffa65fffff] [ 3.903273] debug: unmapping init [mem 0xffffffffa6d14000-0xffffffffa6dfffff] [ 3.938578] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.946885] systemd[1]: Detected virtualization kvm. [ 3.948493] systemd[1]: Detected architecture x86-64. [ 3.950053] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.972462] systemd[1]: No hostname configured. [ 3.974235] systemd[1]: Set hostname to . [ 3.976225] random: systemd: uninitialized urandom read (16 bytes read) [ 3.978786] systemd[1]: Initializing machine ID from random generator. [ 4.025299] random: ln: uninitialized urandom read (6 bytes read) [ 4.123661] random: systemd: uninitialized urandom read (16 bytes read) [ 4.127459] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.133132] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.137755] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.803070] device-mapper: uevent: version 1.0.3 [ 4.805531] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.637982] virtio_net virtio0 ens2: renamed from eth0 [ 5.664725] random: fast init done [ 5.738625] scsi host0: ata_piix [ 5.753816] scsi host1: ata_piix [ 5.756823] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.759973] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.183115] dracut-initqueue[591]: RTNETLINK answers: File exists [ 10.384173] random: crng init done [ 10.385918] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.992487] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.519132] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.889240] SELinux: Disabled at runtime. [ 12.969961] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.978594] systemd[1]: Detected virtualization kvm. [ 12.982842] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.622334] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.626217] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.634521] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.640676] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.645033] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.653259] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.661471] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ 13.861421] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 14.132268] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.509249] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.519625] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.658071] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.673222] EDAC sbridge: Ver: 1.1.2 [ 16.286127] Key type dns_resolver registered [ 16.614142] NFS: Registering the id_resolver key type [ 16.616939] Key type id_resolver registered [ 16.619349] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Started Authorization Manager. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg426-server login: [ 40.237349] libcfs: loading out-of-tree module taints kernel. [ 40.257205] Key type ._llcrypt registered [ 40.258419] Key type .llcrypt registered [ 40.296463] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 47.665637] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 48.221563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 48.231066] alg: No test for adler32 (adler32-zlib) [ 49.213111] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 49.537227] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 51.143118] Key type lgssc registered [ 51.678307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 57.662095] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 70.985058] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 75.514852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 75.530855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 76.673077] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 76.684345] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 76.722694] Lustre: lustre-MDT0000: new disk, initializing [ 76.754488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 76.761522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 78.119293] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 83.157127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 83.191554] Lustre: 6483:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 83.200946] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 83.202423] Lustre: Skipped 1 previous similar message [ 83.233687] Lustre: lustre-MDT0001: new disk, initializing [ 83.252119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 83.260842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 83.264835] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 84.528874] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 86.771857] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 89.762835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 89.895166] Lustre: lustre-OST0000: new disk, initializing [ 89.897456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 89.900384] Lustre: 8414:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 89.919251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 90.797206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 90.802162] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 90.836291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 92.065289] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 98.374943] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 98.421362] Lustre: lustre-OST0001: new disk, initializing [ 98.422800] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 98.425230] Lustre: 9494:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 98.442725] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 100.506856] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 106.472237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 106.475412] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 106.484868] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 107.351927] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 109.943214] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 111.895030] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing check_logdir /tmp/testlogs/ [ 113.356563] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing yml_node [ 114.953311] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 115.804638] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 116.668891] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 117.273864] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Jun 28 03:10:34 EDT 2026 [ 122.672702] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 123.196152] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 137.183859] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 137.184236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 137.188925] Lustre: Skipped 3 previous similar messages [ 137.191771] Lustre: Skipped 2 previous similar messages [ 141.027701] Lustre: server umount lustre-MDT0000 complete [ 142.298559] LustreError: 7431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782630659 with bad export cookie 17929495510352433907 [ 142.300191] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 142.302512] LustreError: 7431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 142.308571] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 142.309171] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 142.317484] Lustre: Skipped 1 previous similar message [ 147.423675] LustreError: 10059:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 147.423792] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 147.431599] LustreError: 10059:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 147.433247] Lustre: Skipped 2 previous similar messages [ 148.488221] Lustre: server umount lustre-MDT0001 complete [ 164.319104] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 164.355100] Lustre: server umount lustre-OST0000 complete [ 171.758367] Lustre: server umount lustre-OST0001 complete [ 173.722237] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 176.568819] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 188.079833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 188.203587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 188.212703] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 188.247508] Lustre: lustre-MDT0000: new disk, initializing [ 188.269038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 188.275645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 189.579700] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 193.697856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.737385] Lustre: 15033:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 193.840967] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 195.133731] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 198.700557] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 200.449308] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 202.626346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 202.759587] Lustre: lustre-OST0000: new disk, initializing [ 202.760918] Lustre: Skipped 1 previous similar message [ 202.762318] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 202.765104] Lustre: Skipped 2 previous similar messages [ 202.767511] Lustre: 17002:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 202.796597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.799248] Lustre: Skipped 1 previous similar message [ 204.471623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 204.474181] Lustre: Skipped 1 previous similar message [ 204.475748] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 204.510314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 204.582286] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 208.134285] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 209.482410] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 209.549595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 210.817631] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 210.886137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 215.009311] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 215.009982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 215.011363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 215.011440] LustreError: Skipped 1 previous similar message [ 215.016735] Lustre: Skipped 1 previous similar message [ 217.711247] Lustre: server umount lustre-OST0000 complete [ 219.615808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 219.619722] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 219.627028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 219.629013] Lustre: Skipped 1 previous similar message [ 225.106811] Lustre: server umount lustre-MDT0000 complete [ 225.247839] LustreError: 15044:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 225.251846] LustreError: 15044:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 226.308288] LustreError: 15828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782630743 with bad export cookie 17929495510352435363 [ 226.311744] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 226.312274] LustreError: 15828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 226.463349] Lustre: server umount lustre-MDT0001 complete [ 227.785581] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 03:12:24 (1782630744) [ 228.594572] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 229.140385] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 03:12:25 (1782630745) [ 231.316236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 231.497129] LustreError: 19406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 231.521324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 232.886978] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 235.807547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 237.288443] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 239.677877] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 241.405768] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 243.703903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 245.835533] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 248.293447] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 261.600628] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 261.603760] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 261.608116] Lustre: Skipped 1 previous similar message [ 261.610563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 261.612956] Lustre: Skipped 2 previous similar messages [ 267.501518] Lustre: server umount lustre-OST0000 complete [ 271.839892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 271.843865] LustreError: Skipped 1 previous similar message [ 271.845958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 271.851856] Lustre: Skipped 1 previous similar message [ 274.845356] Lustre: server umount lustre-MDT0000 complete [ 276.068123] LustreError: 19387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782630793 with bad export cookie 17929495510352436252 [ 276.071679] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 276.073497] LustreError: 19387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 276.202670] Lustre: server umount lustre-MDT0001 complete [ 279.437357] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 280.447618] Key type lgssc unregistered [ 280.583403] LNet: 22954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 280.586256] LNetError: 22954:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 280.597422] LNet: Removed LNI 192.168.204.126@tcp [ 280.935137] Key type .llcrypt unregistered [ 280.936168] Key type ._llcrypt unregistered [ 289.141115] Key type ._llcrypt registered [ 289.142417] Key type .llcrypt registered [ 289.180428] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 289.464634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 289.513790] alg: No test for adler32 (adler32-zlib) [ 290.365873] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 290.457072] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 292.031148] Key type lgssc registered [ 292.361174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 295.063115] loop: module loaded [ 295.081875] loop0: detected capacity change from 0 to 204800000 [ 295.193455] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 295.196882] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 295.309771] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 297.175627] loop0: detected capacity change from 0 to 204800000 [ 297.228786] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 297.233688] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 297.237384] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 298.831876] loop0: detected capacity change from 0 to 204800000 [ 300.531689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 300.538860] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 301.659758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 301.670892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 301.701705] Lustre: lustre-MDT0000: new disk, initializing [ 301.721058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 301.728652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 302.979157] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 306.346757] loop1: detected capacity change from 0 to 204800000 [ 308.147383] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 308.287186] Lustre: lustre-OST07c6: new disk, initializing [ 308.288628] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 308.290353] Lustre: Skipped 1 previous similar message [ 308.292532] Lustre: 26115:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 308.316359] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 310.165224] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 315.902887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 315.907554] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 315.917811] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 317.709567] Lustre: 25102:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 317.712092] Lustre: 25102:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 317.714386] Lustre: 25102:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 317.717558] Lustre: 25102:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 317.720500] Lustre: 25102:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 317.722656] Lustre: 25102:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 330.805251] Lustre: 25101:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 330.809609] Lustre: 25101:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 330.812969] Lustre: 25101:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 330.816560] Lustre: 25101:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 330.820455] Lustre: 25101:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 330.823848] Lustre: 25101:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 334.309739] Lustre: 26123:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 334.314019] Lustre: 26123:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 334.316889] Lustre: 26123:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 334.319678] Lustre: 26123:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 334.322666] Lustre: 26123:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 334.325815] Lustre: 26123:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 334.328846] Lustre: 26123:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 334.331923] Lustre: 26123:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 334.334376] Lustre: 26123:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 334.337720] Lustre: 26123:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 334.340980] Lustre: 26123:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 334.343313] Lustre: 26123:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 337.394401] Lustre: 25103:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 337.396790] Lustre: 25103:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 337.398836] Lustre: 25103:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 337.401482] Lustre: 25103:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 337.403162] Lustre: 25103:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 337.405114] Lustre: 25103:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 337.406866] Lustre: 25103:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 337.409559] Lustre: 25103:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 337.411800] Lustre: 25103:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 337.413696] Lustre: 25103:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 337.415660] Lustre: 25103:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 337.418178] Lustre: 25103:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 343.867734] Lustre: 25101:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 343.871405] Lustre: 25101:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 343.874127] Lustre: 25101:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 343.876770] Lustre: 25101:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 343.879703] Lustre: 25101:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 343.882609] Lustre: 25101:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 343.885607] Lustre: 25101:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 343.888725] Lustre: 25101:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 343.891534] Lustre: 25101:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 343.894464] Lustre: 25101:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 343.897514] Lustre: 25101:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 343.900384] Lustre: 25101:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 353.411138] Lustre: 25102:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 353.414767] Lustre: 25102:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 353.417180] Lustre: 25102:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 353.419152] Lustre: 25102:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 353.421326] Lustre: 25102:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 353.424276] Lustre: 25102:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 353.427455] Lustre: 25102:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 353.431146] Lustre: 25102:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 353.434814] Lustre: 25102:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 353.436786] Lustre: 25102:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 353.439255] Lustre: 25102:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 353.441292] Lustre: 25102:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 394.422350] Lustre: server umount lustre-OST07c6 complete [ 408.126569] Lustre: server umount lustre-MDT0000 complete [ 413.379663] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 414.331777] Key type lgssc unregistered [ 414.455480] LNet: 28694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 414.458516] LNetError: 28694:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 414.467369] LNet: Removed LNI 192.168.204.126@tcp [ 414.771180] Key type .llcrypt unregistered [ 414.772758] Key type ._llcrypt unregistered [ 421.998503] Key type ._llcrypt registered [ 421.999839] Key type .llcrypt registered [ 422.035820] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 422.356855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 422.388249] alg: No test for adler32 (adler32-zlib) [ 423.258704] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 423.346636] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 424.927199] Key type lgssc registered [ 425.294208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 427.741757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 427.747322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 428.923101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 430.224862] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 432.976045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 433.084409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 434.314145] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 436.521697] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 438.198973] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 440.345737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 440.481644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 442.250621] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 444.715223] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 453.794393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 462.818183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 462.818226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 462.820616] LustreError: Skipped 1 previous similar message [ 462.827067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 463.327796] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 463.330086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 463.334749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 464.353233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 464.355993] Lustre: Skipped 1 previous similar message [ 465.704244] Lustre: server umount lustre-OST0000 complete [ 468.959815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 468.961948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 468.966948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 474.592166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 474.594948] Lustre: Skipped 3 previous similar messages [ 481.247083] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 481.300100] Lustre: server umount lustre-MDT0000 complete [ 482.532530] LustreError: 29923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782631000 with bad export cookie 16736665398131331720 [ 482.534868] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 482.537557] LustreError: 29923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 482.676209] Lustre: server umount lustre-MDT0001 complete [ 486.325474] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 487.321508] Key type lgssc unregistered [ 487.444420] LNet: 33693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 487.448488] LNetError: 33693:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 487.456358] LNet: Removed LNI 192.168.204.126@tcp [ 487.758941] Key type .llcrypt unregistered [ 487.760042] Key type ._llcrypt unregistered [ 494.531465] Key type ._llcrypt registered [ 494.532378] Key type .llcrypt registered [ 494.564323] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 494.889981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 494.932837] alg: No test for adler32 (adler32-zlib) [ 495.773600] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 495.855642] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 497.431192] Key type lgssc registered [ 497.761908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 500.245193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 500.251204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 501.433169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 502.809251] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 505.401729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 505.502282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 506.652164] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 508.827671] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 510.515931] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 512.646196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 512.784436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 514.609378] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 516.825690] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 527.143534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 531.424416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 531.426822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.430883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 532.450095] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 532.450686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 532.450869] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 532.459793] Lustre: Skipped 1 previous similar message [ 533.234714] Lustre: server umount lustre-OST0000 complete [ 536.539899] Lustre: server umount lustre-MDT0000 complete [ 536.545430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 536.548430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.552750] LustreError: 35677:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.561332] LustreError: 35677:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 537.568098] LustreError: 34939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 537.573474] LustreError: 34939:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 537.747944] LustreError: 34920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782631055 with bad export cookie 5950202582790254138 [ 537.750329] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 537.751838] LustreError: 34920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 537.900392] Lustre: server umount lustre-MDT0001 complete [ 541.240941] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 542.224464] Key type lgssc unregistered [ 542.340385] LNet: 38489:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 542.343883] LNetError: 38489:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 542.354286] LNet: Removed LNI 192.168.204.126@tcp [ 542.630758] Key type .llcrypt unregistered [ 542.631637] Key type ._llcrypt unregistered [ 551.271392] Key type ._llcrypt registered [ 551.272329] Key type .llcrypt registered [ 551.312226] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 551.630197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 551.680122] alg: No test for adler32 (adler32-zlib) [ 552.539598] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 552.626403] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 554.207161] Key type lgssc registered [ 554.569400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 557.061261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 557.068297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.247933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 559.534170] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 562.532081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.671928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 564.110552] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 566.592668] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 568.452594] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 570.734800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 570.881664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 572.743419] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 574.970804] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 583.203313] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 588.258600] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 588.258609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.260565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 588.268331] Lustre: Skipped 1 previous similar message [ 590.542952] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 590.546132] Lustre: Skipped 2 previous similar messages [ 593.377219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 593.379755] Lustre: Skipped 2 previous similar messages [ 593.743485] Lustre: server umount lustre-MDT0000 complete [ 594.894180] LustreError: 39862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782631112 with bad export cookie 2061826683303795864 [ 594.898197] LustreError: 39862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 594.898248] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 595.655753] LustreError: 42092:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 595.661304] LustreError: 42092:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 595.666754] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [ 598.496045] LustreError: 39877:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 598.496307] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 598.506959] Lustre: Skipped 1 previous similar message [ 599.520165] LustreError: 39907:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 600.775832] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [ 600.779693] Lustre: Skipped 2 previous similar messages [ 604.640281] LustreError: 39876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 604.646026] LustreError: 39876:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 609.247113] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 609.354878] Lustre: server umount lustre-MDT0001 complete [ 635.359190] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 635.447418] Lustre: server umount lustre-OST0000 complete [ 640.624294] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 641.662083] Key type lgssc unregistered [ 641.785409] LNet: 43756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 641.788759] LNetError: 43756:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 641.797453] LNet: Removed LNI 192.168.204.126@tcp [ 642.093747] Key type .llcrypt unregistered [ 642.094878] Key type ._llcrypt unregistered [ 650.472944] Key type ._llcrypt registered [ 650.474580] Key type .llcrypt registered [ 650.513872] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 650.826530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 650.883539] alg: No test for adler32 (adler32-zlib) [ 651.731825] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 651.817127] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 653.407146] Key type lgssc registered [ 653.769323] Lustre: Echo OBD driver; http://www.lustre.org/ [ 656.157958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 656.166041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 657.376761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 658.593124] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 661.147421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 661.264444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 662.440077] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 664.542796] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 666.093750] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 668.088535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 668.211026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 670.050526] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 672.371214] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 681.506633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 686.560552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 686.560788] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 686.563445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 686.565600] Lustre: Skipped 1 previous similar message [ 686.568293] LustreError: Skipped 1 previous similar message [ 687.822976] Lustre: lustre-OST0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 687.825393] Lustre: Skipped 1 previous similar message [ 691.680789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 691.682799] Lustre: Skipped 1 previous similar message [ 696.801599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 696.803472] Lustre: Skipped 2 previous similar messages [ 699.359100] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 699.443033] Lustre: server umount lustre-OST0000 complete [ 712.016287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 712.017385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 712.020433] Lustre: Skipped 1 previous similar message [ 712.023887] Lustre: Skipped 2 previous similar messages [ 718.227591] Lustre: server umount lustre-MDT0000 complete [ 719.321934] LustreError: 45176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782631236 with bad export cookie 9621040411955847083 [ 719.324373] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 719.325092] LustreError: 45176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 725.507110] Lustre: server umount lustre-MDT0001 complete [ 728.677066] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 729.616210] Key type lgssc unregistered [ 729.740270] LNet: 48875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.742338] LNetError: 48875:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 729.748317] LNet: Removed LNI 192.168.204.126@tcp [ 730.024156] Key type .llcrypt unregistered [ 730.025132] Key type ._llcrypt unregistered [ 738.320583] Key type ._llcrypt registered [ 738.322010] Key type .llcrypt registered [ 738.357992] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 738.605556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 738.612162] alg: No test for adler32 (adler32-zlib) [ 739.453446] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 739.532739] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 741.111128] Key type lgssc registered [ 741.434948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 743.728896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 743.734527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 744.903340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 746.117755] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 748.700936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 748.813764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 750.010481] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 752.185486] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 753.833553] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 755.948374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 756.080566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 757.758493] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 759.893281] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 770.409854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 777.335558] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 778.379444] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 789.089860] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-28 3h21m45s [ 789.587186] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 790.734062] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 791.010158] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 791.013850] Lustre: Skipped 2 previous similar messages [ 791.751522] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 791.754032] Lustre: Skipped 3 previous similar messages [ 796.500796] Lustre: server umount lustre-MDT0000 complete [ 797.608284] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 797.895269] LustreError: 50264:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 797.900811] LustreError: 50264:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 799.794752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 799.834705] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 799.921140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 799.946391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 800.934095] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 801.154728] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 802.225564] Lustre: DEBUG MARKER: Wait for df (36385) ... [ 802.713390] Lustre: DEBUG MARKER: done [ 805.344147] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 805.344450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 805.346143] LustreError: Skipped 1 previous similar message [ 805.351146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 805.354038] Lustre: Skipped 4 previous similar messages [ 809.766685] Lustre: server umount lustre-OST0000 complete [ 811.489808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 811.490329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 811.494389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 811.497178] Lustre: Skipped 2 previous similar messages [ 811.500523] Lustre: Skipped 3 previous similar messages [ 816.544505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 816.546879] Lustre: Skipped 2 previous similar messages [ 825.311122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 825.370933] Lustre: server umount lustre-MDT0000 complete [ 826.631819] LustreError: 50248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782631344 with bad export cookie 1889387801707559100 [ 826.634427] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 826.635992] LustreError: 50248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 826.778347] Lustre: server umount lustre-MDT0001 complete [ 830.416489] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 831.444721] Key type lgssc unregistered [ 831.576312] LNet: 55204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 831.578778] LNetError: 55204:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 831.587304] LNet: Removed LNI 192.168.204.126@tcp [ 831.898498] Key type .llcrypt unregistered [ 831.899707] Key type ._llcrypt unregistered [ 845.561865] Key type ._llcrypt registered [ 845.562780] Key type .llcrypt registered [ 845.598343] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 846.013464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 846.026473] alg: No test for adler32 (adler32-zlib) [ 846.870118] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 846.950665] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 848.527190] Key type lgssc registered [ 848.880757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 851.337192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 851.343081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 852.469255] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 852.479221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 852.538475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 853.831932] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 856.461764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 856.492458] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 856.502812] Lustre: 57389:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 856.592869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 857.830791] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 860.111458] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 861.828625] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 864.087933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 864.203663] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 864.270375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 866.262240] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 868.660797] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 868.877932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 877.326516] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-28 3h23m14s [ 877.822637] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 880.037501] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 880.806256] Lustre: DEBUG MARKER: Creating a test file and stat it [ 880.820154] LustreError: 58510:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 901.331649] Lustre: lustre-MDT0000: Client 3bd6bea3-349f-436c-b629-504d5f540101 (at 192.168.204.26@tcp) reconnecting [ 920.863089] LustreError: 58510:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 921.502895] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 922.216832] Lustre: DEBUG MARKER: done [ 925.154735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 925.155996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 925.157199] LustreError: Skipped 1 previous similar message [ 925.160843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 925.163418] Lustre: Skipped 1 previous similar message [ 929.720682] Lustre: server umount lustre-OST0000 complete [ 933.343684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 933.346883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 933.352389] Lustre: Skipped 1 previous similar message [ 933.353937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 934.368075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 934.369481] Lustre: Skipped 2 previous similar messages [ 936.416349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 936.418463] Lustre: Skipped 1 previous similar message [ 937.104449] Lustre: server umount lustre-MDT0000 complete [ 938.298364] LustreError: 57380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782631455 with bad export cookie 9708052082980969897 [ 938.300312] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 938.302388] LustreError: 57380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 938.466899] Lustre: server umount lustre-MDT0001 complete [ 941.732089] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 942.740114] Key type lgssc unregistered [ 942.867341] LNet: 61709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.869301] LNetError: 61709:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 942.878347] LNet: Removed LNI 192.168.204.126@tcp [ 943.190749] Key type .llcrypt unregistered [ 943.191827] Key type ._llcrypt unregistered [ 958.536276] Key type ._llcrypt registered [ 958.537482] Key type .llcrypt registered [ 958.571620] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 958.856796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 958.864387] alg: No test for adler32 (adler32-zlib) [ 959.701274] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 959.778673] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 961.351151] Key type lgssc registered [ 961.665962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 962.696963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 962.703176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 963.826879] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 963.835205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 963.890334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 964.183166] Lustre: Failing over lustre-MDT0000 [ 964.303152] Lustre: server umount lustre-MDT0000 complete [ 966.224255] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 03:24:43 (1782631483) [ 968.220313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 968.400373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 969.535552] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 971.956085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 971.986505] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 971.998147] Lustre: 64486:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 972.079403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 973.328775] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 975.605358] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 977.449423] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 979.651483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 979.755386] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 979.825947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 981.785773] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 984.141769] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 984.590353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 992.249299] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 994.785771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 994.786601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 994.790112] Lustre: Skipped 1 previous similar message [ 999.904410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 999.906438] Lustre: Skipped 4 previous similar messages [ 999.965163] Lustre: server umount lustre-MDT0000 complete [ 1001.129116] LustreError: 66337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782631518 with bad export cookie 12298127563370322204 [ 1001.130830] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1001.132643] LustreError: 66337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1001.274988] Lustre: server umount lustre-MDT0001 complete [ 1002.349351] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1003.806365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1006.636312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1011.167600] LustreError: 63406:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000087e09213 x1869224583658624/t0(0) o250->MGC192.168.204.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1011.349335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1011.372129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1012.691625] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1015.475943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1015.577737] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1015.582699] Lustre: Skipped 1 previous similar message [ 1016.953357] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1019.400784] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1020.703126] Lustre: 63409:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782631522/real 1782631522] req@00000000f0da8dce x1869224583658240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782631538 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1020.901403] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1021.240227] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1021.919134] Lustre: 63410:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782631523/real 1782631523] req@0000000099cf584d x1869224583658368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782631539 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1025.056671] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1025.057749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1025.060392] Lustre: Skipped 1 previous similar message [ 1025.065849] Lustre: Skipped 1 previous similar message [ 1026.056112] Lustre: 63407:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782631527/real 1782631527] req@00000000c177067e x1869224583658496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782631543 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1029.150648] Lustre: server umount lustre-MDT0000 complete [ 1030.114151] LustreError: 68277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1030.119342] LustreError: 68277:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1030.377193] LustreError: 64478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782631547 with bad export cookie 12298127563370324472 [ 1030.379621] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1030.380473] LustreError: 64478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1030.542726] Lustre: server umount lustre-MDT0001 complete [ 1032.638878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1035.711399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1041.375666] LustreError: 63406:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000b832d1e x1869224583677312/t0(0) o250->MGC192.168.204.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1041.524168] LustreError: 66332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1041.546415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1041.548494] Lustre: Skipped 1 previous similar message [ 1041.564664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1042.865843] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1045.605562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1045.692504] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1047.000839] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1049.327980] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1051.097309] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1051.104207] Lustre: 63408:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782631552/real 1782631552] req@000000005baca450 x1869224583677056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782631568 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1051.110959] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1051.112640] Lustre: 63408:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1056.223176] Lustre: 63409:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782631557/real 1782631557] req@00000000bfb1eb46 x1869224583677184/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782631573 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1056.224636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1056.226132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1056.227272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1058.901334] Lustre: server umount lustre-MDT0000 complete [ 1060.116220] LustreError: 64477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782631577 with bad export cookie 12298127563370325844 [ 1060.119370] LustreError: 64477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1060.119784] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1060.268933] Lustre: server umount lustre-MDT0001 complete [ 1062.334226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1063.399816] Lustre: DEBUG MARKER: files compared the same [ 1074.360087] Lustre: server umount lustre-OST0000 complete [ 1079.625567] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1080.611979] Key type lgssc unregistered [ 1080.735441] LNet: 74195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1080.739188] LNetError: 74195:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1080.752406] LNet: Removed LNI 192.168.204.126@tcp [ 1081.061802] Key type .llcrypt unregistered [ 1081.062873] Key type ._llcrypt unregistered [ 1089.662902] Key type ._llcrypt registered [ 1089.663905] Key type .llcrypt registered [ 1089.701385] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1090.060769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1090.095225] alg: No test for adler32 (adler32-zlib) [ 1090.959536] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 1091.051154] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1092.631162] Key type lgssc registered [ 1093.012268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1095.517796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1095.524098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1096.711068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1098.010862] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug +malloc all [ 1100.810713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1100.911814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1102.211283] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug +malloc all [ 1104.535361] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1106.269607] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1108.466327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1108.588692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1110.493252] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug +malloc all [ 1112.902032] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1122.914539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1126.367519] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1126.369944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1126.373161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1128.416455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1128.416934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1128.426229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1132.079085] Lustre: server umount lustre-OST0000 complete [ 1133.535803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1133.536168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1133.539366] Lustre: Skipped 1 previous similar message [ 1133.543430] Lustre: Skipped 2 previous similar messages [ 1138.655556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1138.660438] Lustre: Skipped 1 previous similar message [ 1139.410919] Lustre: server umount lustre-MDT0000 complete [ 1140.609800] LustreError: 75572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782631658 with bad export cookie 17467497917281754945 [ 1140.611074] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1140.615298] LustreError: 75572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1140.759403] Lustre: server umount lustre-MDT0001 complete [ 1144.245145] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1145.202022] Key type lgssc unregistered [ 1145.340287] LNet: 79139:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1145.344469] LNetError: 79139:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1145.354357] LNet: Removed LNI 192.168.204.126@tcp [ 1145.641977] Key type .llcrypt unregistered [ 1145.643077] Key type ._llcrypt unregistered [ 1154.741518] Key type ._llcrypt registered [ 1154.743279] Key type .llcrypt registered [ 1154.780288] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1155.094934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1155.115572] alg: No test for adler32 (adler32-zlib) [ 1155.972222] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 1156.059939] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1157.639118] Key type lgssc registered [ 1158.008919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1160.695592] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1160.702446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1176.159352] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1181.279300] LustreError: 80505:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.126@tcp: failed processing log, type 4: rc = -110 [ 1207.903215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1209.904346] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1212.299545] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1214.899247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1214.939029] Lustre: *** cfs_fail_loc=706, val=0*** [ 1215.128723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1215.152891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1216.609097] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1219.361995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1219.489241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1220.747668] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1222.953864] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1224.569532] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1225.695697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1225.698358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1225.703129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1227.746466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1227.746832] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1231.342358] Lustre: server umount lustre-OST0000 complete [ 1232.864186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1232.869161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1232.871881] Lustre: Skipped 2 previous similar messages [ 1237.984084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1237.986521] Lustre: Skipped 1 previous similar message [ 1238.677616] Lustre: server umount lustre-MDT0000 complete [ 1239.901250] LustreError: 80512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782631757 with bad export cookie 17194533363052162701 [ 1239.904597] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1239.905476] LustreError: 80512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1240.056974] Lustre: server umount lustre-MDT0001 complete [ 1243.552641] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1244.573705] Key type lgssc unregistered [ 1244.714422] LNet: 84117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1244.717358] LNetError: 84117:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1244.725379] LNet: Removed LNI 192.168.204.126@tcp [ 1245.057464] Key type .llcrypt unregistered [ 1245.058852] Key type ._llcrypt unregistered [ 1254.127544] Key type ._llcrypt registered [ 1254.129162] Key type .llcrypt registered [ 1254.166446] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1254.496895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1254.530088] alg: No test for adler32 (adler32-zlib) [ 1255.387501] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 1255.491070] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1257.071213] Key type lgssc registered [ 1257.457147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1259.971032] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1259.978706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1262.324037] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1264.790118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1264.992336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1266.225034] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1268.921855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1269.066795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1270.912714] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1273.671789] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1273.675862] Lustre: MGS: Client 978816d1-ded9-4500-b9cc-252dbee184ab (at 0@lo) reconnecting [ 1273.754234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1273.782425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1274.964242] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1284.064293] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1284.066667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1284.070637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1285.089972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1285.091131] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1285.096442] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1288.750197] Lustre: server umount lustre-OST0000 complete [ 1290.208942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1290.209095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1290.217062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1290.219423] Lustre: Skipped 2 previous similar messages [ 1295.328234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1295.330257] Lustre: Skipped 1 previous similar message [ 1296.132865] Lustre: server umount lustre-MDT0000 complete [ 1297.506547] Lustre: server umount lustre-MDT0001 complete [ 1298.747544] Lustre: server umount MGS complete [ 1302.371637] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 03:30:19 (1782631819) [ 1311.824599] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 1315.159529] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1327.028388] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1328.450236] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1332.416978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1332.554161] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1332.574339] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1332.615073] Lustre: lustre-MDT0001: new disk, initializing [ 1332.641847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1332.653562] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1332.657878] Lustre: 93355:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1332.660758] LustreError: 93355:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1334.004152] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1338.235819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1338.383168] Lustre: lustre-OST0000: new disk, initializing [ 1338.386967] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1338.392713] Lustre: 94121:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1338.421346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1340.424210] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1343.136879] Lustre: 94595:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000c456501f x1869224894039040/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1782631876 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1344.082477] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1346.340574] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1346.344842] Lustre: MGS: Client 490f2fbf-1bf0-42ba-b9aa-596a05a3dfa3 (at 0@lo) reconnecting [ 1346.360715] Lustre: 95166:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1346.369204] LustreError: 93340:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1346.373093] LustreError: 93340:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1346.376251] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1346.411881] Lustre: lustre-MDT0000: new disk, initializing [ 1346.457171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1346.468681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1347.488714] Lustre: 95191:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000b385d19b x1869224894044544/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1782631881 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1347.552606] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1347.833043] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1349.537872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1349.542219] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1349.608192] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1367.008082] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1367.011250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1367.015950] Lustre: Skipped 1 previous similar message [ 1367.017719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1372.470458] Lustre: server umount lustre-OST0000 complete [ 1375.200532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1375.200869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1375.202925] LustreError: Skipped 1 previous similar message [ 1375.207037] Lustre: Skipped 1 previous similar message [ 1375.209057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1375.212684] Lustre: Skipped 4 previous similar messages [ 1379.908761] Lustre: server umount lustre-MDT0000 complete [ 1380.320101] LustreError: 95619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1380.326356] LustreError: 95619:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 1387.401754] Lustre: server umount lustre-MDT0001 complete [ 1388.647661] Lustre: server umount MGS complete [ 1392.398815] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 03:31:49 (1782631909) [ 1398.266976] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1399.320231] Key type lgssc unregistered [ 1399.446424] LNet: 97685:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1399.448638] LNetError: 97685:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1399.454321] LNet: Removed LNI 192.168.204.126@tcp [ 1399.764662] Key type .llcrypt unregistered [ 1399.765901] Key type ._llcrypt unregistered [ 1406.929532] Key type ._llcrypt registered [ 1406.930503] Key type .llcrypt registered [ 1406.967805] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1407.369861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1407.382455] alg: No test for adler32 (adler32-zlib) [ 1408.225614] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 1408.318351] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1409.903144] Key type lgssc registered [ 1410.263723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1412.950318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1412.955963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1414.155438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1415.522563] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1418.425511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1418.550614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1419.925223] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1422.352271] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1424.178414] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1426.401346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1426.561690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1428.524824] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1430.887479] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1438.886271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1444.322091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1444.323384] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1444.323893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1444.326854] Lustre: Skipped 1 previous similar message [ 1444.331152] LustreError: Skipped 1 previous similar message [ 1449.441995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1449.444145] Lustre: Skipped 2 previous similar messages [ 1450.095464] Lustre: server umount lustre-OST0000 complete [ 1454.560889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1454.561193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1454.561932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1454.571437] Lustre: Skipped 1 previous similar message [ 1457.431830] Lustre: server umount lustre-MDT0000 complete [ 1458.679285] LustreError: 101392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782631976 with bad export cookie 7268442819470772466 [ 1458.682419] LustreError: 101392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1458.682614] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1458.827702] Lustre: server umount lustre-MDT0001 complete [ 1462.356657] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1463.383792] Key type lgssc unregistered [ 1463.503463] LNet: 102444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1463.507750] LNetError: 102444:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1463.518349] LNet: Removed LNI 192.168.204.126@tcp [ 1463.827822] Key type .llcrypt unregistered [ 1463.829743] Key type ._llcrypt unregistered [ 1470.130294] Key type ._llcrypt registered [ 1470.131258] Key type .llcrypt registered [ 1470.168666] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1470.504770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1470.571343] alg: No test for adler32 (adler32-zlib) [ 1471.443456] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 1471.547887] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1473.127154] Key type lgssc registered [ 1473.476500] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1475.435886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1475.435996] LustreError: 103504:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1475.440105] Lustre: Skipped 1 previous similar message [ 1475.446375] LustreError: 103505:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1475.449312] LustreError: 103504:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 1475.454206] LustreError: 103504:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1475.458076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1475.458901] LustreError: 103504:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1476.648752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1479.128018] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1479.218993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1480.554892] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1482.172652] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing lsmod [ 1483.147541] LustreError: 104710:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1483.147804] LustreError: 104709:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1483.154336] LustreError: 104710:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 1483.157955] LustreError: 104710:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1483.161401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1483.163361] LustreError: 104710:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1483.344729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1488.353258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1488.354513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1488.357084] Lustre: Skipped 2 previous similar messages [ 1488.358813] Lustre: Skipped 1 previous similar message [ 1489.378463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1489.381723] Lustre: Skipped 3 previous similar messages [ 1490.388122] Lustre: server umount lustre-MDT0000 complete [ 1491.594318] LustreError: 103519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632009 with bad export cookie 2132905284023527177 [ 1491.599226] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1491.599514] LustreError: 103519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1493.471261] LustreError: 103972:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000064695461 x1869225120514048/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1493.553884] Lustre: server umount lustre-MDT0001 complete [ 1505.091163] Lustre: server umount lustre-OST0000 complete [ 1507.937454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1508.146794] LustreError: 105774:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1508.154731] LustreError: 105774:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1508.173810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1509.532746] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1512.396114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1512.516111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1513.876313] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1516.503530] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1518.323349] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1520.631906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1520.788175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1522.761303] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1525.207573] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1534.118750] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1538.527789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1538.530078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1538.533641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1539.554281] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1539.554776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1543.213820] Lustre: server umount lustre-OST0000 complete [ 1544.672542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1544.673099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1544.678655] Lustre: Skipped 1 previous similar message [ 1544.680336] Lustre: Skipped 2 previous similar messages [ 1549.792166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1549.795055] Lustre: Skipped 2 previous similar messages [ 1550.551747] Lustre: server umount lustre-MDT0000 complete [ 1551.804121] LustreError: 105754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632069 with bad export cookie 2132905284023527863 [ 1551.807971] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1551.808915] LustreError: 105754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1551.950230] Lustre: server umount lustre-MDT0001 complete [ 1555.516730] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1556.523213] Key type lgssc unregistered [ 1556.638452] LNet: 109325:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1556.641192] LNetError: 109325:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1556.647302] LNet: Removed LNI 192.168.204.126@tcp [ 1556.961755] Key type .llcrypt unregistered [ 1556.962781] Key type ._llcrypt unregistered [ 1566.191171] Key type ._llcrypt registered [ 1566.192504] Key type .llcrypt registered [ 1566.235206] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1566.631746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1566.660091] alg: No test for adler32 (adler32-zlib) [ 1567.525095] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 1567.622713] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1569.215156] Key type lgssc registered [ 1569.575487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1572.168596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1572.174731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1573.353333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1574.660338] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1577.469253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1577.592441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1578.915500] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1581.359924] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1583.259172] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1585.501770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1585.651529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1587.520136] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1589.814470] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1597.928429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 1602.094557] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1602.828274] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1608.160915] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1608.164355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1608.168256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1608.170181] Lustre: Skipped 1 previous similar message [ 1609.905544] Lustre: server umount lustre-OST0000 complete [ 1613.281080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1613.281609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1613.284671] Lustre: Skipped 2 previous similar messages [ 1613.287535] Lustre: Skipped 1 previous similar message [ 1617.235303] Lustre: server umount lustre-MDT0000 complete [ 1618.400092] LustreError: 112151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1618.404868] LustreError: 112151:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1618.484222] LustreError: 113269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632136 with bad export cookie 12084547357716490883 [ 1618.487511] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1618.489485] LustreError: 113269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1618.641449] Lustre: server umount lustre-MDT0001 complete [ 1622.223617] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1623.266246] Key type lgssc unregistered [ 1623.392368] LNet: 114369:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1623.394591] LNetError: 114369:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1623.401285] LNet: Removed LNI 192.168.204.126@tcp [ 1623.700444] Key type .llcrypt unregistered [ 1623.701922] Key type ._llcrypt unregistered [ 1630.637743] Key type ._llcrypt registered [ 1630.638678] Key type .llcrypt registered [ 1630.678209] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1631.044609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1631.095985] alg: No test for adler32 (adler32-zlib) [ 1631.956754] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 1632.050539] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1633.631166] Key type lgssc registered [ 1633.994664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1636.688986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1636.695523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1637.888045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1639.235388] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1642.081219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1642.206248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1643.596168] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1645.933437] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1647.694872] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1649.982612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1650.134062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1652.068733] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1654.445883] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1663.463196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 1668.576893] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1668.577704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1668.578460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1668.586067] Lustre: Skipped 1 previous similar message [ 1669.935730] Lustre: server umount lustre-OST0000 complete [ 1673.183863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1673.186376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1673.190375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1673.192087] Lustre: Skipped 1 previous similar message [ 1677.269769] Lustre: server umount lustre-MDT0000 complete [ 1678.520545] LustreError: 115549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632196 with bad export cookie 6454343186274790531 [ 1678.524543] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1678.525355] LustreError: 115549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1678.699872] Lustre: server umount lustre-MDT0001 complete [ 1682.538358] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1683.597297] Key type lgssc unregistered [ 1683.727487] LNet: 119118:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.730057] LNetError: 119118:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1683.740332] LNet: Removed LNI 192.168.204.126@tcp [ 1684.065082] Key type .llcrypt unregistered [ 1684.066276] Key type ._llcrypt unregistered [ 1692.922277] Key type ._llcrypt registered [ 1692.924379] Key type .llcrypt registered [ 1692.963810] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1693.297318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1693.304946] alg: No test for adler32 (adler32-zlib) [ 1694.169934] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 1694.263465] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1695.847172] Key type lgssc registered [ 1696.222695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1698.826143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1698.832919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1700.044380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1701.449209] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1704.191210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1704.317200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1705.606297] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1707.861806] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1709.718812] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1711.982188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1712.139048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1714.098971] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1716.471608] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1724.451102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 1729.584052] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1730.782638] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1730.785458] Lustre: Skipped 2 previous similar messages [ 1732.130756] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1732.133307] Lustre: Skipped 2 previous similar messages [ 1736.194913] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1736.208214] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1738.362058] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.26@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1738.365727] Lustre: Skipped 2 previous similar messages [ 1746.424418] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.26@tcp 192.168.0.[2,10]@tcp [ 1748.698957] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.26/24@tcp in log lustre-MDT0000 [ 1748.701411] Lustre: Skipped 2 previous similar messages [ 1754.612704] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.26/24@tcp [ 1754.614333] Lustre: Skipped 1 previous similar message [ 1760.227541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1760.230323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1760.231693] Lustre: Skipped 1 previous similar message [ 1760.232185] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1760.233663] Lustre: Skipped 1 previous similar message [ 1763.699547] Lustre: server umount lustre-OST0000 complete [ 1765.344621] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1765.345327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1765.348468] Lustre: Skipped 1 previous similar message [ 1765.351011] Lustre: Skipped 1 previous similar message [ 1770.464215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1770.466484] Lustre: Skipped 1 previous similar message [ 1771.096744] Lustre: server umount lustre-MDT0000 complete [ 1772.293199] LustreError: 120535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632289 with bad export cookie 15753805689244308228 [ 1772.296562] LustreError: 120535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1772.296860] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1772.470569] Lustre: server umount lustre-MDT0001 complete [ 1775.939242] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1776.981361] Key type lgssc unregistered [ 1777.117338] LNet: 125800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1777.119727] LNetError: 125800:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1777.126321] LNet: Removed LNI 192.168.204.126@tcp [ 1777.442186] Key type .llcrypt unregistered [ 1777.444084] Key type ._llcrypt unregistered [ 1787.536529] Key type ._llcrypt registered [ 1787.537486] Key type .llcrypt registered [ 1787.575590] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1788.027892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1788.039219] alg: No test for adler32 (adler32-zlib) [ 1788.893287] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 1788.995801] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1790.583145] Key type lgssc registered [ 1790.928944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1793.491223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1793.498217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1794.679328] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1794.681462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1794.687841] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.26/24@tcp [ 1795.954372] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1798.663234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1798.775136] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1798.783346] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.26/24@tcp [ 1798.797895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1800.069662] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1802.334444] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1804.123704] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1806.299177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1806.453848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1808.263825] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1810.559031] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1819.748708] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 1824.223862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1824.226299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1824.230086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1825.248496] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1825.251494] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1825.252140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1825.259268] Lustre: Skipped 1 previous similar message [ 1825.913238] Lustre: server umount lustre-OST0000 complete [ 1829.203914] Lustre: server umount lustre-MDT0000 complete [ 1829.855758] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1829.858132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1829.862029] LustreError: 127354:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1829.868658] LustreError: 127354:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1830.368692] LustreError: 127335:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1830.372991] LustreError: 127335:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1830.390079] LustreError: 127318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632348 with bad export cookie 11893159678291434369 [ 1830.393034] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1830.393246] LustreError: 127318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1830.556135] Lustre: server umount lustre-MDT0001 complete [ 1834.073188] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1835.115918] Key type lgssc unregistered [ 1835.240301] LNet: 130940:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1835.242712] LNetError: 130940:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1835.249260] LNet: Removed LNI 192.168.204.126@tcp [ 1835.547720] Key type .llcrypt unregistered [ 1835.548762] Key type ._llcrypt unregistered [ 1843.765444] Key type ._llcrypt registered [ 1843.766652] Key type .llcrypt registered [ 1843.809020] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 1849.662637] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1850.010677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1850.052788] alg: No test for adler32 (adler32-zlib) [ 1850.893135] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 1850.983947] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1852.559113] Key type lgssc registered [ 1852.904485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1863.504877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1863.509797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1864.641122] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1864.651846] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1864.692273] Lustre: lustre-MDT0000: new disk, initializing [ 1864.714747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1864.720603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1865.984432] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1869.960922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1870.003068] Lustre: 134310:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1870.027576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1870.029312] Lustre: Skipped 1 previous similar message [ 1870.067834] Lustre: lustre-MDT0001: new disk, initializing [ 1870.099859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1870.110838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1870.114169] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1871.386352] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1874.991443] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1876.691429] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1878.901189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1879.058421] Lustre: lustre-OST0000: new disk, initializing [ 1879.060653] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1879.065264] Lustre: 136281:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1879.102833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1881.070316] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1884.610142] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1885.907149] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1887.224053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1887.229093] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1887.255523] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1888.007593] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 1889.252340] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1889.317057] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1892.322699] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1892.322947] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1892.323558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1892.325530] LustreError: Skipped 1 previous similar message [ 1892.331911] Lustre: Skipped 1 previous similar message [ 1896.049229] Lustre: server umount lustre-OST0000 complete [ 1897.440286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1897.440858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1897.444040] Lustre: Skipped 1 previous similar message [ 1897.445768] Lustre: Skipped 1 previous similar message [ 1902.560486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1902.564664] Lustre: Skipped 2 previous similar messages [ 1903.387028] Lustre: server umount lustre-MDT0000 complete [ 1904.613353] LustreError: 134303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632422 with bad export cookie 15413312439463571037 [ 1904.616801] LustreError: 134303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1904.617146] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1904.758104] Lustre: server umount lustre-MDT0001 complete [ 1907.430676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1907.634537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1908.939545] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1911.715404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1911.846441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1913.141479] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1915.497411] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1917.243336] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1919.448906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1921.475218] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1923.743283] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1925.086380] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1933.360735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 1934.620552] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1934.689673] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1938.040355] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1938.100504] Lustre: lustre-OST0001: new disk, initializing [ 1938.103113] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1938.106670] Lustre: 141933:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1938.136615] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1938.138787] Lustre: Skipped 1 previous similar message [ 1939.972809] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1943.191510] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1947.645603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1947.648592] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1947.673523] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1948.389393] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 1949.689297] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1949.766287] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1951.465584] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f7f06536800.ost_server_uuid 50 [ 1951.978770] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f7f06536800.ost_server_uuid in FULL state after 0 sec [ 1957.858513] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1957.859940] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1957.860755] LustreError: Skipped 1 previous similar message [ 1957.864762] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1957.867802] Lustre: Skipped 1 previous similar message [ 1959.951156] Lustre: server umount lustre-OST0001 complete [ 1962.982678] LustreError: 142701:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1962.983501] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1962.996312] LustreError: 142701:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1963.004494] Lustre: Skipped 1 previous similar message [ 1963.005104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1963.005414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1967.475723] Lustre: server umount lustre-OST0000 complete [ 1973.217271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1973.217515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1973.218187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1973.218192] Lustre: Skipped 1 previous similar message [ 1973.221104] Lustre: Skipped 2 previous similar messages [ 1974.808724] Lustre: server umount lustre-MDT0000 complete [ 1976.019491] LustreError: 141932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632493 with bad export cookie 15413312439463571905 [ 1976.021826] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1976.023540] LustreError: 141932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1976.175915] Lustre: server umount lustre-MDT0001 complete [ 1982.284067] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1983.280605] Key type lgssc unregistered [ 1983.415311] LNet: 144965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1983.417702] LNetError: 144965:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1983.425303] LNet: Removed LNI 192.168.204.126@tcp [ 1983.714160] Key type .llcrypt unregistered [ 1983.716181] Key type ._llcrypt unregistered [ 1996.327189] Key type ._llcrypt registered [ 1996.328632] Key type .llcrypt registered [ 1996.366113] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 2002.369509] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2002.682285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2002.719066] alg: No test for adler32 (adler32-zlib) [ 2003.572412] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 2003.656091] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2005.239220] Key type lgssc registered [ 2005.596960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2016.490432] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2016.495274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2017.622807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2017.657144] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2017.697859] Lustre: lustre-MDT0000: new disk, initializing [ 2017.719644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2017.726182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2018.978821] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2023.021523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2023.072126] Lustre: 149088:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2023.097273] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2023.099385] Lustre: Skipped 1 previous similar message [ 2023.137882] Lustre: lustre-MDT0001: new disk, initializing [ 2023.169204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2023.180502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2023.183758] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2024.486610] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2028.141378] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2029.909767] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2032.119064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2032.265605] Lustre: lustre-OST0000: new disk, initializing [ 2032.268514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2032.272091] Lustre: 151055:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2032.300790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2032.725067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2032.728085] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2032.758993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2034.178234] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2037.785570] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2039.692381] Lustre: Failing over lustre-OST0000 [ 2039.774263] Lustre: server umount lustre-OST0000 complete [ 2052.576682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2052.730980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2052.737016] Lustre: Skipped 1 previous similar message [ 2052.740066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2052.750029] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2054.350848] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2054.624333] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2055.874196] Lustre: Failing over lustre-MDT0000 [ 2055.971511] Lustre: server umount lustre-MDT0000 complete [ 2058.209043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2058.213838] Lustre: Skipped 2 previous similar messages [ 2059.231164] Lustre: 147319:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782632560/real 1782632560] req@000000000d477a2c x1869225678366208/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1782632576 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2063.328077] LustreError: 149899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2063.333382] LustreError: 149899:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2064.583624] LustreError: 149095:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2065.375087] Lustre: 147317:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782632566/real 1782632566] req@00000000ecff84e3 x1869225678367872/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782632582 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2068.447945] LustreError: 149099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2068.452750] LustreError: 149099:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2068.807777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2068.849623] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2068.853693] Lustre: 149092:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.126@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2068.925138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2068.942689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2068.963863] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2069.703563] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2070.267643] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2074.085476] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2074.092202] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2074.120374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2079.205662] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2079.208965] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2079.210809] LustreError: Skipped 1 previous similar message [ 2079.220151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2079.225609] Lustre: Skipped 1 previous similar message [ 2081.191854] Lustre: server umount lustre-OST0000 complete [ 2084.321185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2084.321219] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2084.321849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2084.324888] Lustre: Skipped 2 previous similar messages [ 2089.440333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2089.442793] Lustre: Skipped 2 previous similar messages [ 2094.560781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2094.562544] Lustre: Skipped 1 previous similar message [ 2097.119126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2097.186985] Lustre: server umount lustre-MDT0000 complete [ 2098.411523] LustreError: 149082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632616 with bad export cookie 8455014288960618549 [ 2098.414948] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2098.416089] LustreError: 149082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2098.585430] Lustre: server umount lustre-MDT0001 complete [ 2102.149561] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2103.171453] Key type lgssc unregistered [ 2103.296537] LNet: 154703:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2103.299571] LNetError: 154703:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2103.307334] LNet: Removed LNI 192.168.204.126@tcp [ 2103.661091] Key type .llcrypt unregistered [ 2103.662792] Key type ._llcrypt unregistered [ 2123.434297] Key type ._llcrypt registered [ 2123.435282] Key type .llcrypt registered [ 2123.475472] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2123.820150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2123.836718] alg: No test for adler32 (adler32-zlib) [ 2124.684322] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 2124.771495] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2126.351120] Key type lgssc registered [ 2126.703119] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2129.409756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2129.418492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.605201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2132.216606] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2135.903760] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2137.858399] LustreError: 156261:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2137.863759] LustreError: 156261:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2137.868623] LustreError: 156261:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2138.028254] Lustre: server umount lustre-MDT0000 complete [ 2142.754675] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 2145.771371] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2157.159563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2157.290344] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2157.299819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2157.345576] Lustre: lustre-MDT0000: new disk, initializing [ 2157.366069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2157.372362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2158.716875] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2162.855184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2162.901712] Lustre: 159558:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2162.930908] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2162.933678] Lustre: Skipped 1 previous similar message [ 2162.972087] Lustre: lustre-MDT0001: new disk, initializing [ 2163.002372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2163.014074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2163.017254] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2164.319402] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2167.921776] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2169.652452] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2171.808644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2171.955386] Lustre: lustre-OST0000: new disk, initializing [ 2171.959117] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2171.963279] Lustre: 161526:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2171.995139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2174.020160] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2176.506114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2176.508752] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2176.547210] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2177.644714] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2179.456676] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2180.006965] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 03:44:56 (1782632696) [ 2239.968365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2239.970809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2239.974206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2243.039635] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2243.039639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2246.247580] Lustre: server umount lustre-MDT0000 complete [ 2247.424627] LustreError: 159550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632765 with bad export cookie 10833918944743379711 [ 2247.425677] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2247.430696] LustreError: 159550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2248.159488] LustreError: 160307:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.159548] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2248.163446] LustreError: 160307:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2248.171074] Lustre: Skipped 1 previous similar message [ 2248.173158] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2248.176515] Lustre: Skipped 2 previous similar messages [ 2252.191447] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2252.191447] LustreError: 162243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2253.640221] Lustre: server umount lustre-MDT0001 complete [ 2269.151134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2269.253189] Lustre: server umount lustre-OST0000 complete [ 2272.645570] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 2275.662413] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2288.016974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2288.140146] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2288.152076] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2288.189252] Lustre: lustre-MDT0000: new disk, initializing [ 2288.209494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2288.215041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2289.483472] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2293.936112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2293.975574] Lustre: 166148:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2293.996688] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2293.998498] Lustre: Skipped 1 previous similar message [ 2294.031962] Lustre: lustre-MDT0001: new disk, initializing [ 2294.072807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2294.076579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2295.364080] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2299.069692] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2300.837778] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2303.505980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2303.648248] Lustre: lustre-OST0000: new disk, initializing [ 2303.649727] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2303.651904] Lustre: 168116:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2303.679036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2303.682229] Lustre: Skipped 1 previous similar message [ 2304.833620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2304.837747] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2304.877391] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2305.841852] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2309.802431] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2311.134406] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2311.205546] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2312.540089] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2312.605746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2315.234793] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2315.235436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2315.235864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2315.241597] Lustre: Skipped 1 previous similar message [ 2315.248810] LustreError: Skipped 1 previous similar message [ 2319.371130] Lustre: server umount lustre-OST0000 complete [ 2324.960150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2324.964176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2324.970519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2324.972952] Lustre: Skipped 1 previous similar message [ 2326.751040] Lustre: server umount lustre-MDT0000 complete [ 2327.992182] LustreError: 166140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632845 with bad export cookie 10833918944743538527 [ 2327.994995] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2327.998343] LustreError: 166140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2328.159944] Lustre: server umount lustre-MDT0001 complete [ 2330.899098] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:47:27 (1782632847) [ 2336.291465] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 2339.855386] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2351.966093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2352.107562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2352.117934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2352.155683] Lustre: lustre-MDT0000: new disk, initializing [ 2352.177698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2352.184808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2353.599465] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2358.453110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.514539] Lustre: 172864:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2358.634761] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2360.056641] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2363.713196] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2365.867303] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2368.075306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2368.210382] Lustre: 174842:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2370.061566] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2370.873451] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2370.909871] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2373.624902] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2376.163683] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2376.164853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2376.166329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2376.166336] Lustre: Skipped 2 previous similar messages [ 2376.168029] LustreError: Skipped 1 previous similar message [ 2376.178185] Lustre: Skipped 2 previous similar messages [ 2381.677266] Lustre: server umount lustre-OST0000 complete [ 2384.351911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2389.231926] Lustre: server umount lustre-MDT0000 complete [ 2390.523159] LustreError: 172856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632908 with bad export cookie 10833918944743539395 [ 2390.524475] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2390.526223] LustreError: 172856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2390.672333] Lustre: server umount lustre-MDT0001 complete [ 2393.720793] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 03:48:30 (1782632910) [ 2400.675366] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 2403.821711] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2414.630022] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2414.748661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2414.757331] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2414.759019] Lustre: Skipped 3 previous similar messages [ 2414.793753] Lustre: lustre-MDT0000: new disk, initializing [ 2414.795186] Lustre: Skipped 2 previous similar messages [ 2414.815846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2414.817736] Lustre: Skipped 2 previous similar messages [ 2414.822151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2414.824573] Lustre: Skipped 2 previous similar messages [ 2415.985812] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2419.772784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2419.812951] Lustre: 179155:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2419.907528] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2421.050486] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2424.527100] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2426.168619] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2428.199064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2428.331316] Lustre: 181129:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2430.069909] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2430.105256] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2433.450985] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2435.552778] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2435.554617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2435.556043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2435.557967] Lustre: Skipped 3 previous similar messages [ 2435.559628] Lustre: Skipped 6 previous similar messages [ 2441.266915] Lustre: server umount lustre-OST0000 complete [ 2448.536627] Lustre: server umount lustre-MDT0000 complete [ 2449.722172] LustreError: 181128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782632967 with bad export cookie 10833918944743540466 [ 2449.725225] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2449.727215] LustreError: 181128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2449.871070] Lustre: server umount lustre-MDT0001 complete [ 2453.054984] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2453.960962] Key type lgssc unregistered [ 2454.079319] LNet: 182980:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2454.081670] LNetError: 182980:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2454.088279] LNet: Removed LNI 192.168.204.126@tcp [ 2454.380552] Key type .llcrypt unregistered [ 2454.381584] Key type ._llcrypt unregistered [ 2462.718321] Key type ._llcrypt registered [ 2462.720143] Key type .llcrypt registered [ 2462.755908] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2463.128333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2463.181635] alg: No test for adler32 (adler32-zlib) [ 2464.051946] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 2464.137672] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2465.719141] Key type lgssc registered [ 2466.048915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2468.540539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2468.545896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2469.721403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2471.061804] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2473.682146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2473.796576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2475.041327] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2477.371059] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2479.007827] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2481.041890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2481.187726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2483.114445] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2485.361992] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2494.499801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2509.800216] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2509.807405] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2509.819682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2509.823494] Lustre: Skipped 1 previous similar message [ 2513.323852] Lustre: server umount lustre-OST0000 complete [ 2514.913232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2514.913573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2514.914240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2514.918463] Lustre: Skipped 2 previous similar messages [ 2514.926944] LustreError: Skipped 1 previous similar message [ 2520.032235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2520.034131] Lustre: Skipped 2 previous similar messages [ 2520.857097] Lustre: server umount lustre-MDT0000 complete [ 2522.021206] LustreError: 184353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782633039 with bad export cookie 10692003504473229988 [ 2522.024866] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2522.024921] LustreError: 184353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2522.171800] Lustre: server umount lustre-MDT0001 complete [ 2525.404059] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2526.295583] Key type lgssc unregistered [ 2526.410299] LNet: 187925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2526.413155] LNetError: 187925:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2526.424275] LNet: Removed LNI 192.168.204.126@tcp [ 2526.702740] Key type .llcrypt unregistered [ 2526.703767] Key type ._llcrypt unregistered [ 2534.851019] Key type ._llcrypt registered [ 2534.852014] Key type .llcrypt registered [ 2534.890877] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2535.232531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2535.269824] alg: No test for adler32 (adler32-zlib) [ 2536.119339] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 2536.215218] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2537.799140] Key type lgssc registered [ 2538.146714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2541.111333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2541.123384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2542.320775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2543.620537] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2546.857580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2546.983078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2548.162037] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2550.714261] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2552.635275] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2555.550913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2555.715540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2557.764555] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2560.242042] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2567.014185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2576.083325] Lustre: lustre-OST0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 2577.380595] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2577.381452] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2577.387682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2577.391630] LustreError: Skipped 1 previous similar message [ 2579.327426] Lustre: server umount lustre-OST0000 complete [ 2582.058588] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9f7f2baac800.ost_server_uuid 50 [ 2582.611433] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2592.721790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2592.722735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2592.725432] Lustre: Skipped 2 previous similar messages [ 2592.727231] Lustre: Skipped 1 previous similar message [ 2597.856594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2597.859048] Lustre: Skipped 1 previous similar message [ 2598.901295] Lustre: server umount lustre-MDT0000 complete [ 2600.324221] LustreError: 189297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782633117 with bad export cookie 12315277631266586595 [ 2600.327015] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2600.328580] LustreError: 189297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2600.498651] Lustre: server umount lustre-MDT0001 complete [ 2603.379130] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 03:52:00 (1782633120) [ 2605.807000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2606.033917] LustreError: 193228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2606.041189] LustreError: 193228:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2606.059569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2607.557669] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2610.697152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2610.829931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2612.169990] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2614.506583] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2616.349221] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2618.530533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2620.721442] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2623.326673] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2625.631421] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2625.718699] Lustre: lustre-OST0001: new disk, initializing [ 2625.720298] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2625.722642] Lustre: 196066:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2625.762125] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2625.763974] Lustre: Skipped 1 previous similar message [ 2628.283201] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2629.006139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2629.009460] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2629.061803] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2631.091231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2632.394661] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2636.263329] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2636.264542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2636.268084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2636.273070] Lustre: Skipped 1 previous similar message [ 2639.368607] Lustre: server umount lustre-OST0000 complete [ 2641.379029] LustreError: 196531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.382937] LustreError: 196531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2641.575353] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2641.645672] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2642.998675] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2643.062740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2643.144995] LustreError: 195049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2646.499721] LustreError: 196531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2646.507536] LustreError: 196531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2651.621867] LustreError: 195051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2651.627063] LustreError: 195051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2654.689059] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2654.689343] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2654.692947] LustreError: Skipped 2 previous similar messages [ 2654.693761] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2654.699503] Lustre: Skipped 1 previous similar message [ 2654.706617] Lustre: Skipped 3 previous similar messages [ 2659.396127] Lustre: server umount lustre-OST0001 complete [ 2661.857185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2661.862947] Lustre: Skipped 1 previous similar message [ 2667.098847] Lustre: server umount lustre-MDT0000 complete [ 2668.290076] LustreError: 196065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782633185 with bad export cookie 12315277631266587652 [ 2668.292242] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2668.293628] LustreError: 196065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2668.448070] Lustre: server umount lustre-MDT0001 complete [ 2675.808342] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 03:53:12 (1782633192) [ 2677.888212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2678.058511] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2678.071211] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2678.149783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2679.461852] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2682.400352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2682.438976] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2682.457806] Lustre: 199481:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2683.891981] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2686.192280] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2687.854812] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2689.965288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2690.065331] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2691.885040] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2693.702306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 2694.122853] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2696.227486] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2696.251093] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2698.084217] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2700.212216] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2701.795590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2701.796767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2701.801983] Lustre: Skipped 1 previous similar message [ 2701.805524] Lustre: Skipped 6 previous similar messages [ 2707.066308] Lustre: server umount lustre-OST0000 complete [ 2710.729017] LustreError: 201328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2710.733709] LustreError: 201328:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2722.272593] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2722.274636] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2722.277629] Lustre: Skipped 1 previous similar message [ 2723.688726] Lustre: server umount lustre-OST0001 complete [ 2737.632061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2737.635530] Lustre: Skipped 10 previous similar messages [ 2739.167119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2739.244142] Lustre: server umount lustre-MDT0000 complete [ 2740.557570] LustreError: 199471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782633258 with bad export cookie 12315277631266589038 [ 2740.562096] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2740.562896] LustreError: 199471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2740.740288] Lustre: server umount lustre-MDT0001 complete [ 2747.779622] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 03:54:24 (1782633264) [ 2752.402397] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 2754.938123] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2765.486965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2765.608807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2765.617773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2765.655890] Lustre: lustre-MDT0000: new disk, initializing [ 2765.675190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2765.677041] Lustre: Skipped 3 previous similar messages [ 2765.681439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2766.887826] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2770.855531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2770.896066] Lustre: 207542:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2770.919982] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2770.922541] Lustre: Skipped 1 previous similar message [ 2770.963835] Lustre: lustre-MDT0001: new disk, initializing [ 2771.008142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2771.011833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2772.300950] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2775.822760] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2777.464057] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2779.585633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2779.726150] Lustre: lustre-OST0000: new disk, initializing [ 2779.727591] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2779.729771] Lustre: 209509:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2781.610222] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2783.158907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2783.164118] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2783.190213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2785.154242] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2786.409956] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2786.472553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2787.701402] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2787.768521] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2790.443359] Lustre: server umount lustre-OST0000 complete [ 2793.440860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2793.444179] Lustre: Skipped 3 previous similar messages [ 2801.659049] Lustre: server umount lustre-MDT0000 complete [ 2802.848237] LustreError: 207534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782633320 with bad export cookie 12315277631266590557 [ 2802.851928] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2802.854294] LustreError: 207534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2812.462106] Lustre: server umount lustre-MDT0001 complete [ 2815.119272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2815.310610] LustreError: 211725:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2815.315990] LustreError: 211725:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2816.577037] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2819.211082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2820.572947] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2822.720243] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2824.327674] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2826.379597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2828.209463] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2830.378440] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2831.626826] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2840.849827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2842.164149] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2842.245153] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2844.127885] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2844.130171] LustreError: Skipped 1 previous similar message [ 2844.132384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2844.133937] Lustre: Skipped 4 previous similar messages [ 2848.945073] Lustre: server umount lustre-OST0000 complete [ 2850.811330] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2852.098753] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2875.504244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2875.643444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2875.647197] Lustre: Skipped 5 previous similar messages [ 2877.425855] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2879.747196] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2880.244203] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2883.242235] Lustre: server umount lustre-OST0000 complete [ 2884.374863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2884.378057] Lustre: Skipped 3 previous similar messages [ 2886.485217] Lustre: server umount lustre-MDT0000 complete [ 2887.768306] LustreError: 211706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782633405 with bad export cookie 12315277631266591425 [ 2887.771911] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2887.773098] LustreError: 211706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2887.909877] Lustre: server umount lustre-MDT0001 complete [ 2890.519447] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 03:56:47 (1782633407) [ 2892.604338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2892.797721] LustreError: 217741:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2894.279367] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2897.079104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2898.419901] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2900.746756] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2902.488253] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2904.590854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2906.436178] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2908.687882] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2910.038957] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2919.298335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2920.500824] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2920.564539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2922.596845] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2922.660986] Lustre: lustre-OST0001: new disk, initializing [ 2922.662650] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2922.664817] Lustre: 221175:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2924.488434] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2928.021438] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2929.351225] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2931.198545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2931.201420] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2931.226489] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2931.468448] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2932.729710] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2932.794391] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2936.291611] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2936.294784] LustreError: Skipped 2 previous similar messages [ 2939.517282] Lustre: server umount lustre-OST0001 complete [ 2941.422803] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2941.486881] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2942.810930] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2942.881879] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2958.537474] LustreError: 222140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2958.544918] LustreError: 222140:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 2966.336740] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2968.390948] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2970.768199] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2971.328044] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2971.622492] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 2971.625906] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 2971.625948] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2971.630881] Lustre: Skipped 1 previous similar message [ 2973.902599] Lustre: lustre-OST0001: Not available for connect from 192.168.204.26@tcp (stopping) [ 2973.906214] Lustre: Skipped 6 previous similar messages [ 2976.737017] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2976.739132] LustreError: Skipped 1 previous similar message [ 2978.041729] Lustre: server umount lustre-OST0001 complete [ 2996.330850] Lustre: server umount lustre-OST0000 complete [ 3003.743641] Lustre: server umount lustre-MDT0000 complete [ 3004.962222] LustreError: 221173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782633522 with bad export cookie 12315277631266592356 [ 3004.964617] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3004.966131] LustreError: 221173:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3019.231162] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3019.356341] Lustre: server umount lustre-MDT0001 complete [ 3027.258570] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 03:59:04 (1782633544) [ 3029.447401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3029.603160] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3029.613977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3029.686668] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3029.688569] Lustre: Skipped 5 previous similar messages [ 3031.054824] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3033.865433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3033.898954] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3033.910330] Lustre: 226423:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3035.331315] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3037.869751] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3039.884124] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3042.220077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3042.322182] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3044.480330] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3047.126744] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3056.315423] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3056.340327] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3058.389834] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3060.659294] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3062.264577] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 3063.513217] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3063.580585] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3064.941373] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3065.026168] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3066.980397] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9f7f0382c800.ost_server_uuid 50 [ 3067.500605] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9f7f0382c800.ost_server_uuid in FULL state after 0 sec [ 3067.810942] Lustre: Permanently deactivating lustre-OST0001 [ 3067.815597] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3068.588755] Lustre: Permanently reactivating lustre-OST0001 [ 3073.507355] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3073.508939] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3073.509540] LustreError: Skipped 3 previous similar messages [ 3073.515379] Lustre: Skipped 8 previous similar messages [ 3075.644287] Lustre: server umount lustre-OST0001 complete [ 3082.985493] Lustre: server umount lustre-OST0000 complete [ 3090.262848] Lustre: server umount lustre-MDT0000 complete [ 3091.408360] LustreError: 226414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782633609 with bad export cookie 12315277631266593630 [ 3091.412043] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3091.413070] LustreError: 226414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3091.554382] Lustre: server umount lustre-MDT0001 complete [ 3098.650200] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 04:00:15 (1782633615) [ 3100.942147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3101.073820] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3101.085076] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3101.087828] Lustre: Skipped 5 previous similar messages [ 3102.399378] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3105.008291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3105.041025] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3105.050493] Lustre: 233099:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3106.401956] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3108.654444] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3110.293193] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3112.400068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3112.513733] Lustre: Permanently deactivating lustre-OST0000 [ 3112.517869] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3112.547356] LustreError: 234945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3112.547671] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3112.554129] LustreError: 234945:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 3114.404808] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3116.645290] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3118.757852] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3120.696183] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3122.621940] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3122.659181] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3122.921422] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3123.433399] Lustre: Permanently reactivating lustre-OST0000 [ 3132.407962] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3132.410335] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3132.413745] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3132.432947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3138.020132] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3138.023717] LustreError: Skipped 2 previous similar messages [ 3140.604020] Lustre: server umount lustre-OST0001 complete [ 3147.955778] Lustre: server umount lustre-OST0000 complete [ 3155.354543] Lustre: server umount lustre-MDT0000 complete [ 3156.568182] LustreError: 233091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782633674 with bad export cookie 12315277631266595247 [ 3156.570834] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3156.571344] LustreError: 233091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3156.722027] Lustre: server umount lustre-MDT0001 complete [ 3160.175362] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3161.164625] Key type lgssc unregistered [ 3161.286376] LNet: 237980:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3161.290302] LNetError: 237980:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3161.301290] LNet: Removed LNI 192.168.204.126@tcp [ 3161.614381] Key type .llcrypt unregistered [ 3161.615451] Key type ._llcrypt unregistered [ 3169.501113] Key type ._llcrypt registered [ 3169.502158] Key type .llcrypt registered [ 3169.535835] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3169.862580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3169.874142] alg: No test for adler32 (adler32-zlib) [ 3170.724904] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 3170.809784] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3172.383193] Key type lgssc registered [ 3172.731055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3175.818514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3175.825949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3177.012954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3178.314891] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3181.052445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3181.090854] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3181.099931] Lustre: 239426:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3181.103600] Lustre: Permanently deactivating lustre-MDT0001 [ 3181.108360] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3181.214604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3181.932404] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3181.934449] LustreError: 239468:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3181.937804] LustreError: 239468:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3181.942404] LustreError: 239468:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 1, failed: rc = -108 [ 3182.455265] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3184.670502] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3186.434731] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3188.540656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3188.694386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3190.466028] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3190.756766] LustreError: 241276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3190.759868] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3190.763024] LustreError: 241276:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3192.600048] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3194.629689] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3194.696018] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3196.422590] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3198.698892] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3199.974619] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3202.024599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3202.024669] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3204.710219] Lustre: Permanently reactivating lustre-MDT0001 [ 3204.717429] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3204.719344] Lustre: Skipped 2 previous similar messages [ 3206.287128] Lustre: lustre-MDT0001: Client a5dd8c68-4c22-4fe3-adb7-5a1192c5eb9d (at 192.168.204.26@tcp) reconnecting [ 3208.462364] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3208.972698] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3210.619681] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3210.691923] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3211.998766] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3212.070366] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3212.915966] Lustre: Permanently deactivating lustre-MDT0001 [ 3212.922205] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3212.924060] Lustre: Skipped 2 previous similar messages [ 3221.502966] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3226.529500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3226.530812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3226.532835] Lustre: Skipped 2 previous similar messages [ 3226.535588] Lustre: Skipped 3 previous similar messages [ 3230.812682] Lustre: server umount lustre-MDT0000 complete [ 3231.713500] LustreError: 239436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3231.721713] LustreError: 239436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3231.999059] LustreError: 239417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782633749 with bad export cookie 1781223722491421461 [ 3232.002217] LustreError: 239417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3232.002682] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3232.143148] Lustre: server umount lustre-MDT0001 complete [ 3239.419096] Lustre: server umount lustre-OST0000 complete [ 3243.103460] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 04:02:39 (1782633759) [ 3246.449421] Lustre: server umount lustre-OST0001 complete [ 3248.342616] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 3251.176252] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3262.536733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3262.654886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3262.657850] Lustre: Skipped 2 previous similar messages [ 3262.665355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3262.703548] Lustre: lustre-MDT0000: new disk, initializing [ 3262.722325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3262.727285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3263.984615] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3268.118921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3268.164756] Lustre: 247699:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3268.191886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3268.194066] Lustre: Skipped 1 previous similar message [ 3268.230041] Lustre: lustre-MDT0001: new disk, initializing [ 3268.281856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3268.286084] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3269.648121] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3273.410795] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3275.172023] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3277.674100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3277.837270] Lustre: lustre-OST0000: new disk, initializing [ 3277.839281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3277.843621] Lustre: 249668:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3277.872086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3277.874274] Lustre: Skipped 1 previous similar message [ 3280.147566] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3283.609143] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3283.956638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3283.959512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3283.989385] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3289.237261] LustreError: 247704:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3293.201777] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3293.264533] Lustre: lustre-OST0001: new disk, initializing [ 3293.267395] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3293.271644] Lustre: 250781:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3293.826222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3293.829108] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3293.870312] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3295.102655] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3298.597553] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3299.102646] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3299.327051] LustreError: 247704:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3299.334706] LustreError: 250789:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3309.423065] LustreError: 250789:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3312.335775] Lustre: lustre-OST0001: Not available for connect from 192.168.204.26@tcp (stopping) [ 3314.658330] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3314.661038] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3314.661706] LustreError: Skipped 1 previous similar message [ 3314.664738] Lustre: Skipped 1 previous similar message [ 3314.665565] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3314.669814] Lustre: Skipped 1 previous similar message [ 3316.280714] Lustre: server umount lustre-OST0001 complete [ 3319.780519] LustreError: 249653:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3319.784762] LustreError: 249653:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3324.898678] LustreError: 249652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3324.903438] LustreError: 249652:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3330.019258] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3330.021675] LustreError: 251658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3330.023496] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3330.024854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3330.034735] LustreError: 251658:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3330.041349] Lustre: Skipped 2 previous similar messages [ 3334.316323] Lustre: server umount lustre-OST0000 complete [ 3340.257495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3340.257628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3340.258722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3340.258728] Lustre: Skipped 1 previous similar message [ 3340.263065] LustreError: Skipped 1 previous similar message [ 3340.277544] Lustre: Skipped 1 previous similar message [ 3341.818565] Lustre: server umount lustre-MDT0000 complete [ 3342.963457] LustreError: 248493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782633860 with bad export cookie 1781223722491424247 [ 3342.968054] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3342.969737] LustreError: 248493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3349.193511] Lustre: server umount lustre-MDT0001 complete [ 3352.439986] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3353.360393] Key type lgssc unregistered [ 3353.469384] LNet: 252835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3353.472829] LNetError: 252835:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3353.486318] LNet: Removed LNI 192.168.204.126@tcp [ 3353.766553] Key type .llcrypt unregistered [ 3353.767598] Key type ._llcrypt unregistered [ 3366.975201] Key type ._llcrypt registered [ 3366.976303] Key type .llcrypt registered [ 3367.009562] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3367.430205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3367.437767] alg: No test for adler32 (adler32-zlib) [ 3368.285645] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 3368.375611] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3369.951124] Key type lgssc registered [ 3370.312682] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3372.750648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3372.756260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3373.875941] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3373.883675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3373.938807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3375.168676] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3377.830329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3377.861827] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3377.873035] Lustre: 255018:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3377.961803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3379.204190] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3381.473747] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3383.153709] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3385.290556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3385.415244] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3385.483103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3387.486331] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3388.495892] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3389.700543] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3429.345269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3429.346120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3429.347449] LustreError: Skipped 1 previous similar message [ 3429.352989] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3429.356276] Lustre: Skipped 1 previous similar message [ 3434.289381] Lustre: server umount lustre-OST0000 complete [ 3435.665595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3439.675141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3439.850062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3441.580808] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3443.640810] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3450.083633] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3450.087349] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3450.087430] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3450.093519] Lustre: Skipped 1 previous similar message [ 3481.057976] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3481.058538] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3481.061802] LustreError: Skipped 1 previous similar message [ 3481.066683] Lustre: Skipped 1 previous similar message [ 3481.067227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3481.072593] Lustre: Skipped 1 previous similar message [ 3482.421865] Lustre: server umount lustre-OST0000 complete [ 3485.663763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3485.665810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3485.668980] Lustre: Skipped 1 previous similar message [ 3485.670523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3489.691975] Lustre: server umount lustre-MDT0000 complete [ 3490.830087] LustreError: 255007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782634008 with bad export cookie 52843238924431897 [ 3490.832663] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3490.833180] LustreError: 255007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3490.987401] Lustre: server umount lustre-MDT0001 complete [ 3494.272603] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3495.185906] Key type lgssc unregistered [ 3495.316429] LNet: 260428:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3495.320308] LNetError: 260428:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3495.329316] LNet: Removed LNI 192.168.204.126@tcp [ 3495.612588] Key type .llcrypt unregistered [ 3495.613640] Key type ._llcrypt unregistered [ 3503.871640] Key type ._llcrypt registered [ 3503.872608] Key type .llcrypt registered [ 3503.907629] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3504.168941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3504.176769] alg: No test for adler32 (adler32-zlib) [ 3505.028295] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 3505.110310] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3506.687070] Key type lgssc registered [ 3507.016102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3509.441748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3509.447573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3510.630798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3511.887154] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3514.450968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3514.565162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3515.740837] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3517.854793] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3519.461650] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3521.539440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3521.687596] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3523.355750] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3525.542958] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3536.038389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3545.568313] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3545.571780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3545.577809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3546.592736] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3546.594587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3546.595126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3549.097379] Lustre: server umount lustre-OST0000 complete [ 3550.687825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3550.689821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3550.693170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3550.694681] Lustre: Skipped 1 previous similar message [ 3556.430559] Lustre: server umount lustre-MDT0000 complete [ 3556.832102] LustreError: 261820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3556.835931] LustreError: 261820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3557.580385] LustreError: 262555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782634075 with bad export cookie 18175120042200101594 [ 3557.583794] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3557.585589] LustreError: 262555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3557.726635] Lustre: server umount lustre-MDT0001 complete [ 3560.898512] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3561.841433] Key type lgssc unregistered [ 3561.959348] LNet: 266139:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3561.961501] LNetError: 266139:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3561.970292] LNet: Removed LNI 192.168.204.126@tcp [ 3562.255685] Key type .llcrypt unregistered [ 3562.257183] Key type ._llcrypt unregistered [ 3568.971922] Key type ._llcrypt registered [ 3568.973832] Key type .llcrypt registered [ 3569.013134] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3569.443379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3569.449536] alg: No test for adler32 (adler32-zlib) [ 3570.295907] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 3570.384496] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3571.967120] Key type lgssc registered [ 3572.317191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3574.677737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3574.683756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3575.877647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3577.190837] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3579.998038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3580.125771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3581.467078] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3583.744886] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3585.514870] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3587.653520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3587.792880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3589.544395] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3591.763035] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3602.151912] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3612.644741] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3612.644973] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3612.647338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3612.649840] Lustre: Skipped 1 previous similar message [ 3612.654327] Lustre: Skipped 1 previous similar message [ 3616.746864] Lustre: server umount lustre-OST0000 complete [ 3621.343863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3621.346950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3621.352343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3622.880064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3622.880501] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3628.000414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3628.003100] Lustre: Skipped 2 previous similar messages [ 3632.095139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3632.164876] Lustre: server umount lustre-MDT0000 complete [ 3633.121805] LustreError: 267336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3633.127478] LustreError: 267336:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3633.359543] LustreError: 267317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782634150 with bad export cookie 3296065261454985197 [ 3633.363897] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3633.535176] Lustre: server umount lustre-MDT0001 complete [ 3636.871148] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3637.778615] Key type lgssc unregistered [ 3637.897355] LNet: 271035:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3637.899578] LNetError: 271035:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3637.907290] LNet: Removed LNI 192.168.204.126@tcp [ 3638.185639] Key type .llcrypt unregistered [ 3638.186756] Key type ._llcrypt unregistered [ 3647.051227] Key type ._llcrypt registered [ 3647.052350] Key type .llcrypt registered [ 3647.086225] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3647.381573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3647.425074] alg: No test for adler32 (adler32-zlib) [ 3648.277715] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 3648.371361] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3649.951158] Key type lgssc registered [ 3650.304257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3652.703167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3652.708361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3653.881644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3655.126528] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3657.799780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3657.928579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3659.257713] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3661.578904] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3663.229856] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3665.301930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3665.445811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3667.270405] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3669.489990] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3678.757620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3687.903893] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3687.906041] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3687.909653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3688.929118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3688.930715] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3688.931935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3692.905689] Lustre: server umount lustre-OST0000 complete [ 3699.168836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3699.169084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3699.169659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3699.169664] Lustre: Skipped 1 previous similar message [ 3699.172465] Lustre: Skipped 1 previous similar message [ 3704.288202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3704.290226] Lustre: Skipped 2 previous similar messages [ 3708.383098] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3708.446944] Lustre: server umount lustre-MDT0000 complete [ 3709.408982] LustreError: 272428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3709.413113] LustreError: 272428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3709.588199] LustreError: 272410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782634227 with bad export cookie 3771369948330183786 [ 3709.591245] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3709.593629] LustreError: 272410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3709.731886] Lustre: server umount lustre-MDT0001 complete [ 3712.996515] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3713.876222] Key type lgssc unregistered [ 3714.003352] LNet: 276753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3714.006347] LNetError: 276753:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3714.014298] LNet: Removed LNI 192.168.204.126@tcp [ 3714.295111] Key type .llcrypt unregistered [ 3714.296153] Key type ._llcrypt unregistered [ 3720.826995] Key type ._llcrypt registered [ 3720.827954] Key type .llcrypt registered [ 3720.859752] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3721.205608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3721.234673] alg: No test for adler32 (adler32-zlib) [ 3722.075654] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 3722.162087] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3723.735124] Key type lgssc registered [ 3724.038764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3726.365244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3726.370430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3727.554508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3728.781821] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3731.452725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3731.564962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3732.770347] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3734.932071] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3736.513470] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3738.511815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3738.654143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3740.357888] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3742.566943] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3752.998463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3757.024067] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3757.026163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3757.029748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3758.049405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3758.049758] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3758.054588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3762.666973] Lustre: server umount lustre-OST0000 complete [ 3767.263893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3767.266089] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3767.269816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3767.272635] Lustre: Skipped 1 previous similar message [ 3773.408083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3773.409839] Lustre: Skipped 3 previous similar messages [ 3778.015048] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3778.074871] Lustre: server umount lustre-MDT0000 complete [ 3778.527944] LustreError: 277976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3778.532201] LustreError: 277976:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3779.263019] LustreError: 278725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782634296 with bad export cookie 5501685986558574438 [ 3779.265899] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3779.266059] LustreError: 278725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3779.411552] Lustre: server umount lustre-MDT0001 complete [ 3782.746819] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3783.598739] Key type lgssc unregistered [ 3783.716335] LNet: 281683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3783.718359] LNetError: 281683:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3783.728325] LNet: Removed LNI 192.168.204.126@tcp [ 3784.008035] Key type .llcrypt unregistered [ 3784.008982] Key type ._llcrypt unregistered [ 3787.467521] Key type ._llcrypt registered [ 3787.468797] Key type .llcrypt registered [ 3787.510761] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 3792.998150] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 3798.953629] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3799.285705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3799.326130] alg: No test for adler32 (adler32-zlib) [ 3800.169250] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 3800.264308] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3801.839153] Key type lgssc registered [ 3802.171038] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3812.725150] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3812.730046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3813.867627] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3813.881333] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3813.927996] Lustre: lustre-MDT0000: new disk, initializing [ 3813.949676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3813.955312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3815.164915] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3819.097420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3819.147142] Lustre: 285208:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3819.170956] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3819.174541] Lustre: Skipped 1 previous similar message [ 3819.217573] Lustre: lustre-MDT0001: new disk, initializing [ 3819.248361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3819.259442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3819.262697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3820.522841] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3824.025377] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3825.748369] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3827.882941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3828.021456] Lustre: lustre-OST0000: new disk, initializing [ 3828.023139] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3828.026237] Lustre: 287176:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3828.053571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3829.860122] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3831.159104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3831.162080] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3831.192368] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3833.455052] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3834.799672] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3834.862402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3836.121979] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3836.194380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3841.507042] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3841.507294] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3841.508167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3841.509256] LustreError: Skipped 1 previous similar message [ 3841.512750] Lustre: Skipped 1 previous similar message [ 3846.626813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3846.629105] Lustre: Skipped 2 previous similar messages [ 3851.231111] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3851.322536] Lustre: server umount lustre-OST0000 complete [ 3855.327778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3855.329773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3855.333177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3856.864836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3858.842998] Lustre: server umount lustre-MDT0000 complete [ 3860.172274] LustreError: 287175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782634377 with bad export cookie 768271002074461111 [ 3860.175229] LustreError: 287175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3860.175970] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3860.318498] Lustre: server umount lustre-MDT0001 complete [ 3862.998261] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 04:12:59 (1782634379) [ 3865.174661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3865.377416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3866.563207] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3869.169263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3870.516889] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3872.699320] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3874.331937] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3876.409957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3876.543096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3876.545164] Lustre: Skipped 1 previous similar message [ 3878.367913] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3880.675371] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3894.239748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3894.242067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3894.245628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3894.247729] Lustre: Skipped 2 previous similar messages [ 3900.214737] Lustre: server umount lustre-OST0000 complete [ 3904.991764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3904.992935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3904.993840] LustreError: Skipped 2 previous similar messages [ 3904.997956] Lustre: Skipped 1 previous similar message [ 3904.999428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3905.001885] Lustre: Skipped 2 previous similar messages [ 3907.614736] Lustre: server umount lustre-MDT0000 complete [ 3908.718125] LustreError: 289581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782634426 with bad export cookie 768271002074461979 [ 3908.720537] LustreError: 290336:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 3908.721249] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3908.722770] LustreError: 289581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3908.875224] Lustre: server umount lustre-MDT0001 complete [ 3912.179136] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3913.172887] Key type lgssc unregistered [ 3913.288306] LNet: 293151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3913.290388] LNetError: 293151:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3913.302272] LNet: Removed LNI 192.168.204.126@tcp [ 3913.579193] Key type .llcrypt unregistered [ 3913.580187] Key type ._llcrypt unregistered [ 3922.396595] Key type ._llcrypt registered [ 3922.398021] Key type .llcrypt registered [ 3922.432789] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 3927.929433] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3928.291657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3928.326019] alg: No test for adler32 (adler32-zlib) [ 3929.175985] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 3929.275957] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3930.855108] Key type lgssc registered [ 3931.148907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3944.577578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3944.583070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3945.700839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3945.711324] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3945.749155] Lustre: lustre-MDT0000: new disk, initializing [ 3945.768876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3945.773978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3947.148320] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3951.102173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3951.138555] Lustre: 296946:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3951.161696] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3951.163244] Lustre: Skipped 1 previous similar message [ 3951.195710] Lustre: lustre-MDT0001: new disk, initializing [ 3951.225454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3951.239505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3951.242646] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3952.371298] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3955.827939] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3957.556923] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3959.523771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3959.649539] Lustre: lustre-OST2710: new disk, initializing [ 3959.650977] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 3959.653118] Lustre: 298913:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3959.678671] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 3961.375678] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3964.856828] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 3964.931487] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 3964.936112] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 3964.969726] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 3967.960461] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3968.053004] Lustre: lustre-OST03e8: new disk, initializing [ 3968.056665] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 3968.062173] Lustre: 299976:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3968.108305] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 3970.225655] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3971.588378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 3971.592083] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 3971.627874] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 3972.737562] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 3975.272209] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 3975.337604] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3976.605623] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 3976.666906] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3977.992266] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 3978.055647] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3979.314377] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 3979.376434] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3981.797423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3981.800827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3981.805108] Lustre: Skipped 3 previous similar messages [ 3981.809085] Lustre: Skipped 2 previous similar messages [ 3986.394711] Lustre: server umount lustre-MDT0000 complete [ 3987.678159] LustreError: 296983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782634505 with bad export cookie 18029507660114495837 [ 3987.681431] LustreError: 296983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3987.682108] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3987.812858] Lustre: server umount lustre-MDT0001 complete [ 3999.235777] Lustre: server umount lustre-OST2710 complete [ 4009.653512] Lustre: server umount lustre-OST03e8 complete [ 4015.192667] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 4018.096613] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4029.078337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4029.199700] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4029.211649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4029.253561] Lustre: lustre-MDT0000: new disk, initializing [ 4029.273237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4029.278145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4030.430019] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4034.548473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4034.591560] Lustre: 305214:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4034.692032] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4035.943858] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4039.490267] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4041.047174] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4043.433667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4043.558058] Lustre: lustre-OST0000: new disk, initializing [ 4043.559286] Lustre: Skipped 1 previous similar message [ 4043.560657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4043.562399] Lustre: Skipped 2 previous similar messages [ 4043.564150] Lustre: 307182:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4043.588518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4043.590260] Lustre: Skipped 1 previous similar message [ 4045.261278] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4048.851530] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4049.918138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4049.920879] Lustre: Skipped 1 previous similar message [ 4049.922952] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4049.955454] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4050.308461] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4050.366976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4051.696524] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4051.776257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4055.008589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4055.009372] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4055.010628] LustreError: Skipped 1 previous similar message [ 4055.014309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4055.017278] Lustre: Skipped 2 previous similar messages [ 4058.548294] Lustre: server umount lustre-OST0000 complete [ 4060.128203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4060.128942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4060.131610] Lustre: Skipped 2 previous similar messages [ 4060.134567] Lustre: Skipped 1 previous similar message [ 4065.248266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4065.251224] Lustre: Skipped 1 previous similar message [ 4066.224965] Lustre: server umount lustre-MDT0000 complete [ 4067.509292] LustreError: 305208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782634585 with bad export cookie 18029507660114497503 [ 4067.513476] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4067.515972] LustreError: 305208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4067.668548] Lustre: server umount lustre-MDT0001 complete [ 4070.376223] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 04:16:27 (1782634587) [ 4070.829714] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4071.354739] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 04:16:28 (1782634588) [ 4072.361796] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4081.258823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4081.383814] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4081.391415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4081.442664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4082.576716] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4085.221070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4085.249377] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4085.257961] Lustre: 311028:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4086.749320] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4102.625759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4102.721600] Lustre: Denying initial registration attempt from nid 192.168.204.126@tcp, specified as failover [ 4102.724238] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4102.726611] LustreError: 312484:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 4102.728616] LustreError: 312484:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4102.730266] LustreError: 312484:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4102.782303] Lustre: server umount lustre-OST0000 complete [ 4102.783541] LustreError: 312484:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4105.695811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4105.697948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4105.701412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4117.471974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4117.473681] Lustre: Skipped 5 previous similar messages [ 4119.007122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4119.061712] Lustre: server umount lustre-MDT0000 complete [ 4120.163191] LustreError: 311023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782634637 with bad export cookie 18029507660114498371 [ 4120.165616] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4120.166274] LustreError: 311023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4120.301788] Lustre: server umount lustre-MDT0001 complete [ 4123.400395] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 4124.297984] Key type lgssc unregistered [ 4124.403302] LNet: 313605:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4124.405553] LNetError: 313605:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4124.411294] LNet: Removed LNI 192.168.204.126@tcp [ 4124.703624] Key type .llcrypt unregistered [ 4124.704603] Key type ._llcrypt unregistered [ 4127.848447] Key type ._llcrypt registered [ 4127.849349] Key type .llcrypt registered [ 4127.885371] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4128.154530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4128.161334] alg: No test for adler32 (adler32-zlib) [ 4129.005576] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 4129.085449] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 4130.663092] Key type lgssc registered [ 4130.983415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4153.427485] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4155.956724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4155.964946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4157.091924] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4157.098030] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4157.148273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4158.276505] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4160.837713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4160.869369] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4160.879807] Lustre: 316345:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4160.965543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4162.162899] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4164.253849] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4165.847879] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4180.898028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4180.999645] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4181.063542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4182.785775] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4184.931423] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4185.674743] Lustre: server umount lustre-OST0000 complete [ 4188.143921] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4188.145152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4188.147697] Lustre: Skipped 1 previous similar message [ 4188.151290] Lustre: Skipped 1 previous similar message [ 4192.980948] Lustre: server umount lustre-MDT0000 complete [ 4194.131244] LustreError: 316335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782634711 with bad export cookie 5167605343918848661 [ 4194.134334] LustreError: 316335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4194.134794] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4194.280720] Lustre: server umount lustre-MDT0001 complete [ 4197.391324] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 4198.228484] Key type lgssc unregistered [ 4198.340380] LNet: 319904:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4198.343391] LNetError: 319904:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4198.353263] LNet: Removed LNI 192.168.204.126@tcp [ 4198.633520] Key type .llcrypt unregistered [ 4198.634513] Key type ._llcrypt unregistered [ 4206.837354] Key type ._llcrypt registered [ 4206.838350] Key type .llcrypt registered [ 4206.871927] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4207.275565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4207.294569] alg: No test for adler32 (adler32-zlib) [ 4208.135906] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 4208.214117] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 4209.791128] Key type lgssc registered [ 4210.095770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4212.409421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4212.414553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4213.588503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4214.786911] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4217.365151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4217.478481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4218.677120] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4220.773407] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4222.355440] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4226.402057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4226.542376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4228.301778] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4230.460616] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4242.816496] LustreError: 323893:0:(ldlm_resource.c:1207:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200001b71:0x1:0x0].0xf14dc48e (00000000d57bc8f0) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 4243.423771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4243.425897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4243.429229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4243.937229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4243.938462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4243.940768] Lustre: Skipped 1 previous similar message [ 4243.943594] Lustre: Skipped 1 previous similar message [ 4246.221748] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 4248.983691] Lustre: server umount lustre-MDT0000 complete [ 4249.056267] LustreError: 321314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4249.060843] LustreError: 321314:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4250.106560] LustreError: 321277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782634767 with bad export cookie 6759668701028605309 [ 4250.109785] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4250.112531] LustreError: 321277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4251.336106] LustreError: 321292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4251.340899] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [ 4254.175937] LustreError: 321291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4254.176141] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4256.331275] Lustre: server umount lustre-MDT0001 complete [ 4258.378929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4260.983654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4274.655600] LustreError: 320607:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d8ede637 x1869227989426944/t0(0) o250->MGC192.168.204.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4274.797865] LustreError: 323126:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4274.802753] LustreError: 323126:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4274.819461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4274.835208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4275.950322] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4278.514193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4279.652539] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4279.824390] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4281.947671] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4282.424746] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4283.997731] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4284.772987] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4292.625872] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 8 sec [ 4294.112114] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4294.116471] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4294.123179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4294.124698] Lustre: Skipped 2 previous similar messages [ 4300.144726] Lustre: server umount lustre-OST0000 complete [ 4304.352438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4304.355264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4304.359536] Lustre: Skipped 1 previous similar message [ 4304.361607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4304.363725] Lustre: Skipped 2 previous similar messages [ 4307.566667] Lustre: server umount lustre-MDT0000 complete [ 4308.730555] LustreError: 321277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782634826 with bad export cookie 6759668701028612463 [ 4308.735591] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4308.736099] LustreError: 321277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4308.898723] Lustre: server umount lustre-MDT0001 complete [ 4312.160652] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 4313.113994] Key type lgssc unregistered [ 4313.232328] LNet: 327576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4313.235155] LNetError: 327576:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4313.244302] LNet: Removed LNI 192.168.204.126@tcp [ 4313.549222] Key type .llcrypt unregistered [ 4313.550660] Key type ._llcrypt unregistered [ 4321.728762] Key type ._llcrypt registered [ 4321.729653] Key type .llcrypt registered [ 4321.760572] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4322.056392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4322.080384] alg: No test for adler32 (adler32-zlib) [ 4322.919469] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 4323.001252] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 4324.575098] Key type lgssc registered [ 4324.910097] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4327.296359] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4327.303286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4328.482092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4329.671473] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4332.384861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4332.504925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4333.669540] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4335.798573] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4337.435128] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4342.496948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4342.500725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4342.503417] Lustre: Skipped 1 previous similar message [ 4344.533272] Lustre: server umount lustre-MDT0000 complete [ 4345.629189] LustreError: 328949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782634863 with bad export cookie 10934819549177750329 [ 4345.632334] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4345.632383] LustreError: 328949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4345.778776] Lustre: server umount lustre-MDT0001 complete [ 4350.079374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4350.219936] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4350.229666] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4350.298105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4351.442447] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4353.905874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4353.935754] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4353.944841] Lustre: 331552:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4354.028245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4355.127087] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4357.253973] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4358.862121] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4363.233682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4363.332178] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4363.335128] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4363.337505] LustreError: 333432:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 4363.339418] LustreError: 333432:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4363.341053] LustreError: 333432:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4363.393996] Lustre: server umount lustre-OST0000 complete [ 4363.395272] LustreError: 333432:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4367.905808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4368.011105] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4368.077610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4369.804787] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4371.944532] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4373.535051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4374.293104] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4374.327122] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4374.366538] Lustre: lustre-OST0001: new disk, initializing [ 4374.368063] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4374.370203] Lustre: 334969:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4376.156833] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4378.329704] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4379.214270] Lustre: server umount lustre-OST0001 complete [ 4383.712369] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4383.712798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4383.714458] LustreError: Skipped 1 previous similar message [ 4383.717572] Lustre: Skipped 1 previous similar message [ 4383.719427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4383.721194] Lustre: Skipped 1 previous similar message [ 4383.727304] LustreError: 333898:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4383.731336] LustreError: 333898:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4386.474315] Lustre: server umount lustre-OST0000 complete [ 4388.832865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4388.833540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4388.836350] Lustre: Skipped 1 previous similar message [ 4388.838951] Lustre: Skipped 1 previous similar message [ 4393.685825] Lustre: server umount lustre-MDT0000 complete [ 4393.952039] LustreError: 332295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4393.956471] LustreError: 332295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4394.761413] LustreError: 331542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782634912 with bad export cookie 10934819549177750875 [ 4394.764025] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4394.764716] LustreError: 331542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4394.906670] Lustre: server umount lustre-MDT0001 complete [ 4398.168210] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 4399.049399] Key type lgssc unregistered [ 4399.179450] LNet: 336921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4399.181692] LNetError: 336921:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4399.191409] LNet: Removed LNI 192.168.204.126@tcp [ 4399.499864] Key type .llcrypt unregistered [ 4399.500900] Key type ._llcrypt unregistered [ 4429.166454] Key type ._llcrypt registered [ 4429.167571] Key type .llcrypt registered [ 4429.206966] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 4435.394335] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4435.913344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4435.924371] alg: No test for adler32 (adler32-zlib) [ 4436.779242] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 4436.869781] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 4438.455133] Key type lgssc registered [ 4438.803210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4449.400218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4449.405034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4450.518787] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4450.530450] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4450.564276] Lustre: lustre-MDT0000: new disk, initializing [ 4450.583531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4450.588682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4451.827408] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4455.768199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4455.806188] Lustre: 341823:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4455.831322] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4455.832841] Lustre: Skipped 1 previous similar message [ 4455.868704] Lustre: lustre-MDT0001: new disk, initializing [ 4455.903882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4455.917552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4455.920734] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4457.214327] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4460.582171] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4462.231162] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4464.294109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4464.421861] Lustre: lustre-OST0000: new disk, initializing [ 4464.423483] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4464.425740] Lustre: 343786:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4464.453057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4466.245944] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4469.628139] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4470.884108] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4471.802341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4471.805099] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4471.830250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4471.959247] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4473.127564] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4473.187323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4476.898115] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4476.898379] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4476.899846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4476.901221] LustreError: Skipped 1 previous similar message [ 4476.904306] Lustre: Skipped 1 previous similar message [ 4479.916183] Lustre: server umount lustre-OST0000 complete [ 4481.503798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4481.505802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4481.509338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4481.510937] Lustre: Skipped 1 previous similar message [ 4487.136286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4487.137946] Lustre: Skipped 3 previous similar messages [ 4492.256062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4492.257719] Lustre: Skipped 1 previous similar message [ 4495.327081] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4495.383535] Lustre: server umount lustre-MDT0000 complete [ 4496.870053] LustreError: 341815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782635014 with bad export cookie 17013447289632418522 [ 4496.874348] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4496.877832] LustreError: 341815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4497.058779] Lustre: server umount lustre-MDT0001 complete [ 4499.767427] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 04:23:36 (1782635016) [ 4502.456828] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 04:23:39 (1782635019) [ 4505.865741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4506.145194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4507.639590] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4510.161981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4510.266860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4511.397138] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4513.510861] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4515.101175] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4517.494307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4519.738437] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4522.083152] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4535.678585] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4536.173184] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4536.664331] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4537.181290] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4537.749281] Lustre: Failing over lustre-MDT0000 [ 4538.002832] Lustre: server umount lustre-MDT0000 complete [ 4540.895832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4540.897847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4540.900771] Lustre: Skipped 1 previous similar message [ 4540.902230] LustreError: 346545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4540.906046] LustreError: 346545:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4541.408363] LustreError: 349034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4541.413751] LustreError: 349034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4542.669995] LustreError: 346542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4546.528096] LustreError: 346564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4546.535018] LustreError: 346564:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4550.620928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4550.658696] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4550.739447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4550.742591] Lustre: Skipped 1 previous similar message [ 4550.765723] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4551.888441] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4552.904036] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4555.749765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4555.757672] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4555.772128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4557.427556] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4557.976750] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4559.784505] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4560.381906] Lustre: Failing over lustre-MDT0000 [ 4560.499877] Lustre: server umount lustre-MDT0000 complete [ 4560.864557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4560.865752] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4560.866888] LustreError: 346545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4560.869525] Lustre: Skipped 4 previous similar messages [ 4560.874049] LustreError: 346545:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4571.105075] LustreError: 346545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4571.109663] LustreError: 346545:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4573.037947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4573.074503] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4573.154606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4573.175767] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4573.384148] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4574.301900] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4578.277031] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4578.278797] Lustre: Skipped 2 previous similar messages [ 4578.285721] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4578.299777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4579.894839] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4580.419810] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4583.393182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4583.394347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4583.394931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4583.399573] Lustre: Skipped 1 previous similar message [ 4588.687460] Lustre: server umount lustre-MDT0000 complete [ 4589.748062] LustreError: 346525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782635107 with bad export cookie 17013447289632421175 [ 4589.751246] LustreError: 346525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4589.751573] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4589.883209] Lustre: server umount lustre-MDT0001 complete [ 4601.266811] Lustre: server umount lustre-OST0000 complete [ 4605.713929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4605.890366] LustreError: 352895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4605.912828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4607.094580] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4609.637900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4610.865580] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4612.998105] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4614.698830] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4616.684911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4618.456286] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4620.537514] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4631.141853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 4635.616300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4635.618659] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4635.621648] Lustre: Skipped 1 previous similar message [ 4635.623157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4635.625595] Lustre: Skipped 5 previous similar messages [ 4638.438623] Lustre: server umount lustre-OST0000 complete [ 4645.651672] Lustre: server umount lustre-MDT0000 complete [ 4646.730174] LustreError: 354736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782635164 with bad export cookie 17013447289632421924 [ 4646.733436] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4646.735970] LustreError: 354736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4646.872863] Lustre: server umount lustre-MDT0001 complete [ 4649.997883] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 4650.819710] Key type lgssc unregistered [ 4650.935340] LNet: 356446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4650.937335] LNetError: 356446:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4650.948294] LNet: Removed LNI 192.168.204.126@tcp [ 4651.211996] Key type .llcrypt unregistered [ 4651.213068] Key type ._llcrypt unregistered [ 4659.715067] Key type ._llcrypt registered [ 4659.716491] Key type .llcrypt registered [ 4659.747453] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4660.121087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4660.133838] alg: No test for adler32 (adler32-zlib) [ 4660.979601] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 4661.059455] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 4662.631096] Key type lgssc registered [ 4662.936766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4665.128266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4665.134270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4666.305350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4667.428112] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4669.838250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4669.947108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4671.066103] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4673.102218] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4674.606277] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4676.533730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4676.666951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4678.308104] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4680.641764] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4691.046925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 4695.520145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4695.522321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4695.526079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4696.545276] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4696.546450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4696.551368] Lustre: Skipped 2 previous similar messages [ 4700.882549] Lustre: server umount lustre-MDT0000 complete [ 4701.666181] LustreError: 357940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4701.671046] LustreError: 357940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4701.926458] LustreError: 357921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782635219 with bad export cookie 489000060846024945 [ 4701.929656] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4701.931079] LustreError: 357921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4702.064928] Lustre: server umount lustre-MDT0001 complete [ 4713.395017] Lustre: server umount lustre-OST0000 complete [ 4717.214349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4717.397240] LustreError: 361665:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4717.417865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4718.558925] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4721.069941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4722.322778] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4724.320796] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4725.820573] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4727.771983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4727.914732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4727.916429] Lustre: Skipped 1 previous similar message [ 4728.933499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4729.511957] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4731.508523] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4741.600723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4741.602992] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4741.603781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4741.608482] Lustre: Skipped 3 previous similar messages [ 4744.144638] Lustre: server umount lustre-MDT0000 complete [ 4745.188128] LustreError: 361691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782635262 with bad export cookie 489000060846026044 [ 4745.190958] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4745.191163] LustreError: 361691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4745.322411] Lustre: server umount lustre-MDT0001 complete [ 4756.661243] Lustre: server umount lustre-OST0000 complete [ 4760.978635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4761.152047] LustreError: 365573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4761.171527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4762.264329] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4764.674852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4765.884155] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4767.900433] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4769.402189] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4771.330020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4773.055229] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4775.070514] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4785.825922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4795.360082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4795.362172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4795.365588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4795.367058] Lustre: Skipped 2 previous similar messages [ 4797.672406] Lustre: server umount lustre-OST0000 complete [ 4800.479782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4800.481742] LustreError: Skipped 1 previous similar message [ 4800.482838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4800.485705] Lustre: Skipped 1 previous similar message [ 4800.487097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4800.488565] Lustre: Skipped 2 previous similar messages [ 4811.744109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4811.745799] Lustre: Skipped 5 previous similar messages [ 4813.279055] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4813.338480] Lustre: server umount lustre-MDT0000 complete [ 4814.420049] LustreError: 367415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782635332 with bad export cookie 489000060846027052 [ 4814.423177] LustreError: 367415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4814.423288] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4814.560525] Lustre: server umount lustre-MDT0001 complete [ 4817.598821] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 4818.472086] Key type lgssc unregistered [ 4818.589318] LNet: 369125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4818.592269] LNetError: 369125:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4818.604284] LNet: Removed LNI 192.168.204.126@tcp [ 4818.870543] Key type .llcrypt unregistered [ 4818.871613] Key type ._llcrypt unregistered [ 4827.109537] Key type ._llcrypt registered [ 4827.110511] Key type .llcrypt registered [ 4827.142175] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4827.421298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4827.430598] alg: No test for adler32 (adler32-zlib) [ 4828.266531] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 4828.345781] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 4829.919116] Key type lgssc registered [ 4830.228095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4832.474338] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4832.480273] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4832.483542] LustreError: 370541:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 4832.491859] LustreError: 370541:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 4832.494577] LustreError: 370541:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 4832.497398] LustreError: 370541:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 4832.500523] LustreError: 370541:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4834.683641] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4834.686735] LustreError: 370937:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 4834.696854] LustreError: 370937:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 4834.698717] LustreError: 370937:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 4834.700627] LustreError: 370937:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 4834.702732] LustreError: 370937:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4840.165080] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 4840.991674] Key type lgssc unregistered [ 4841.104306] LNet: 371937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4841.106262] LNetError: 371937:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4841.115284] LNet: Removed LNI 192.168.204.126@tcp [ 4841.363662] Key type .llcrypt unregistered [ 4841.364644] Key type ._llcrypt unregistered [ 4846.376074] Key type ._llcrypt registered [ 4846.377089] Key type .llcrypt registered [ 4846.408339] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 4851.883290] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4852.165943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4852.207023] alg: No test for adler32 (adler32-zlib) [ 4853.049111] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 4853.126706] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 4854.703196] Key type lgssc registered [ 4855.105562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4868.726222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4868.733397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4869.858882] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4869.870193] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4869.913188] Lustre: lustre-MDT0000: new disk, initializing [ 4869.938266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4869.944973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4871.374867] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4876.576892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4876.652309] Lustre: 375067:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4876.714373] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4876.719902] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4876.723569] Lustre: Skipped 1 previous similar message [ 4876.767766] Lustre: lustre-MDT0001: new disk, initializing [ 4876.803517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4876.820893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4876.825624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4878.671131] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4882.810549] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4885.153348] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4888.456339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4888.669399] Lustre: lustre-OST0000: new disk, initializing [ 4888.672307] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4888.676269] Lustre: 377039:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4888.716493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4888.759658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4888.762506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4888.802021] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4891.266189] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4896.109216] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4898.167108] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4898.295845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4900.041523] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4900.161207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4904.420125] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4904.420579] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4904.425039] LustreError: Skipped 1 previous similar message [ 4904.433674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4904.439386] Lustre: Skipped 1 previous similar message [ 4907.292692] Lustre: server umount lustre-OST0000 complete [ 4909.537316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4909.538430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4909.545762] Lustre: Skipped 2 previous similar messages [ 4909.551932] Lustre: Skipped 1 previous similar message [ 4914.656134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4914.658009] Lustre: Skipped 1 previous similar message [ 4919.776342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4919.779478] Lustre: Skipped 1 previous similar message [ 4923.359117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4923.456214] Lustre: server umount lustre-MDT0000 complete [ 4925.191978] LustreError: 375061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782635442 with bad export cookie 15818286333076340982 [ 4925.198640] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4925.199797] LustreError: 375061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4925.393916] Lustre: server umount lustre-MDT0001 complete [ 4928.562222] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 4929.161755] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 04:30:45 (1782635445) [ 4932.046948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4932.320602] LustreError: 379496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4932.332850] LustreError: 379496:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4932.361504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4934.005402] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4937.696951] LustreError: 379497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4937.961261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4938.134889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4939.725135] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4943.006353] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4945.309411] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4948.529360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4948.722134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4951.320975] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4954.805104] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4958.050302] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4958.148707] Lustre: lustre-OST0001: new disk, initializing [ 4958.151241] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4958.155027] Lustre: 382329:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4958.803667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4958.809939] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4958.858659] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4960.635256] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4965.380692] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4969.444140] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4969.449109] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4969.452638] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4969.456564] Lustre: Skipped 1 previous similar message [ 4969.463790] Lustre: Skipped 1 previous similar message [ 4973.580508] Lustre: server umount lustre-OST0001 complete [ 4974.566890] LustreError: 381317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4974.576553] LustreError: 381317:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4979.687395] LustreError: 381316:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4979.699333] LustreError: 381316:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4984.804926] LustreError: 381315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4984.814612] LustreError: 381315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4987.872375] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4987.877884] LustreError: Skipped 1 previous similar message [ 4987.881113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4987.890945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4987.894717] Lustre: Skipped 1 previous similar message [ 4989.926179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4995.042810] LustreError: 383262:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4995.048015] LustreError: 383262:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5000.159084] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5000.272268] Lustre: server umount lustre-OST0000 complete [ 5004.768516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5004.773529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5004.780854] Lustre: Skipped 1 previous similar message [ 5008.443251] Lustre: server umount lustre-MDT0000 complete [ 5010.145345] LustreError: 379476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782635527 with bad export cookie 15818286333076341864 [ 5010.148260] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5010.149890] LustreError: 379476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5016.377290] Lustre: server umount lustre-MDT0001 complete [ 5020.964955] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5022.433373] Key type lgssc unregistered [ 5022.591603] LNet: 384317:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5022.596386] LNetError: 384317:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5022.607589] LNet: Removed LNI 192.168.204.126@tcp [ 5023.090522] Key type .llcrypt unregistered [ 5023.092743] Key type ._llcrypt unregistered [ 5046.567571] Key type ._llcrypt registered [ 5046.569122] Key type .llcrypt registered [ 5046.630107] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5047.068101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5047.081587] alg: No test for adler32 (adler32-zlib) [ 5047.929439] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 5048.015951] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5049.591186] Key type lgssc registered [ 5050.036769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5053.226719] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5053.235806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5054.390301] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5054.398034] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5054.474332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5056.035660] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5059.996779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5060.033126] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5060.046052] Lustre: 387088:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5060.154497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5061.847119] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5065.187361] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5067.709455] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5070.303962] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5070.308229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5070.314785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5073.377069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5073.378220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5074.842116] Lustre: server umount lustre-MDT0000 complete [ 5076.553117] LustreError: 387125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782635594 with bad export cookie 14618731137188080597 [ 5076.560420] LustreError: 387125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5076.560474] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5076.727782] Lustre: server umount lustre-MDT0001 complete [ 5080.714742] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 04:33:17 (1782635597) [ 5083.514190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5083.709068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5084.812255] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5087.476330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5087.640956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5088.933419] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5090.957527] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5092.842077] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5094.737305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5094.833263] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5094.885513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5096.787934] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5098.840744] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5108.057838] Lustre: Permanently deactivating lustre-OST0000 [ 5108.062233] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5109.359223] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5109.862874] LustreError: 392440:0:(mgs_llog.c:2041:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5109.868055] LustreError: 392440:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 5109.871441] LustreError: 392440:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5110.421790] LustreError: 392488:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 5110.423816] LustreError: 392488:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5117.149927] Lustre: server umount lustre-OST0000 complete [ 5118.022228] LustreError: 392737:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 5118.024399] LustreError: 392737:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5121.504654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5121.505119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5121.508032] Lustre: Skipped 1 previous similar message [ 5121.512106] Lustre: Skipped 2 previous similar messages [ 5126.624986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5126.627041] Lustre: Skipped 1 previous similar message [ 5128.693536] Lustre: server umount lustre-MDT0000 complete [ 5129.772069] LustreError: 392290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782635647 with bad export cookie 14618731137188081164 [ 5129.774964] LustreError: 392290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5129.775659] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5140.054810] Lustre: server umount lustre-MDT0001 complete [ 5142.941721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5144.790908] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5145.894560] LustreError: 393893:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5147.731479] Lustre: 394231:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.126@tcp [ 5148.471887] Lustre: server umount MGS complete [ 5152.890323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5153.163346] LustreError: 394955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5153.169786] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5153.171965] LustreError: 394955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5153.175381] Lustre: Skipped 1 previous similar message [ 5153.200839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5154.992648] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5157.789334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5157.931413] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5159.731289] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5162.134657] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5164.263575] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5165.192386] Lustre: Permanently reactivating lustre-OST0000 [ 5165.201682] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5165.205962] Lustre: Skipped 2 previous similar messages [ 5189.087176] LustreError: 396685:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5191.709029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5191.857915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5191.859618] Lustre: Skipped 1 previous similar message [ 5193.606971] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5196.027262] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5209.570332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5209.572360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5209.572654] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5209.577597] Lustre: Skipped 1 previous similar message [ 5210.776412] Lustre: server umount lustre-OST0000 complete [ 5214.176387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5214.179784] LustreError: Skipped 1 previous similar message [ 5214.181636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5214.629687] Lustre: server umount lustre-MDT0000 complete [ 5214.689616] LustreError: 394952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5216.315194] LustreError: 394981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782635733 with bad export cookie 14618731137188082200 [ 5216.321878] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5216.323909] LustreError: 394981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5216.529311] Lustre: server umount lustre-MDT0001 complete [ 5221.303649] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5222.641979] Key type lgssc unregistered [ 5222.806110] LNet: 399526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5222.810234] LNetError: 399526:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5222.826530] LNet: Removed LNI 192.168.204.126@tcp [ 5223.222431] Key type .llcrypt unregistered [ 5223.224520] Key type ._llcrypt unregistered [ 5234.763391] Key type ._llcrypt registered [ 5234.765437] Key type .llcrypt registered [ 5234.842070] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 5244.443578] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5244.939556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5244.975184] alg: No test for adler32 (adler32-zlib) [ 5245.919684] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 5246.048481] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5247.655211] Key type lgssc registered [ 5248.189837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5261.708020] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5261.713882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5262.848454] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5262.861096] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5262.899319] Lustre: lustre-MDT0000: new disk, initializing [ 5262.934262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5262.941835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5264.401112] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5268.845029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5268.889725] Lustre: 402668:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5268.913827] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5268.916049] Lustre: Skipped 1 previous similar message [ 5268.954695] Lustre: lustre-MDT0001: new disk, initializing [ 5268.992374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5269.009561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5269.019065] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5270.530730] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5274.492373] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5276.394920] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5278.931751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5279.083574] Lustre: lustre-OST0000: new disk, initializing [ 5279.085237] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5279.087631] Lustre: 404636:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5279.120426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5281.386114] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5285.375886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5285.380572] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5285.436089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5285.595909] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5287.265404] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5287.361762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5288.913173] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5289.003044] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5290.468417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5290.469236] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5290.470672] LustreError: Skipped 1 previous similar message [ 5290.475199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5290.478087] Lustre: Skipped 1 previous similar message [ 5291.840450] Lustre: server umount lustre-OST0000 complete [ 5294.559834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5294.562212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5294.565549] Lustre: Skipped 1 previous similar message [ 5294.567170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5295.584769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5295.585818] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5299.290418] Lustre: server umount lustre-MDT0000 complete [ 5300.504246] LustreError: 403464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782635818 with bad export cookie 2446876261990520288 [ 5300.506779] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5300.508775] LustreError: 403464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5300.685521] Lustre: server umount lustre-MDT0001 complete [ 5303.574463] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5304.128990] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 04:37:00 (1782635820) [ 5306.275035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5306.474519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5307.716313] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5310.429208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5310.553415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5311.798822] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5314.033226] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5315.845245] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5318.176340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5320.253423] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5322.599553] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5323.238227] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5334.321620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5337.057748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5337.057754] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5337.061276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5337.064147] Lustre: Skipped 1 previous similar message [ 5337.067816] Lustre: Skipped 1 previous similar message [ 5342.177185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5342.179343] Lustre: Skipped 1 previous similar message [ 5342.257836] Lustre: server umount lustre-OST0000 complete [ 5346.271841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5346.274528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5349.719038] Lustre: server umount lustre-MDT0000 complete [ 5350.897108] LustreError: 407072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782635868 with bad export cookie 2446876261990521142 [ 5350.899081] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5350.901589] LustreError: 407072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5351.044958] Lustre: server umount lustre-MDT0001 complete [ 5354.737049] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5355.728862] Key type lgssc unregistered [ 5355.858399] LNet: 410832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5355.861819] LNetError: 410832:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5355.869324] LNet: Removed LNI 192.168.204.126@tcp [ 5356.155762] Key type .llcrypt unregistered [ 5356.156875] Key type ._llcrypt unregistered [ 5363.505283] Key type ._llcrypt registered [ 5363.506855] Key type .llcrypt registered [ 5363.543263] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5363.642355] Key type .llcrypt unregistered [ 5363.643643] Key type ._llcrypt unregistered [ 5370.489277] Key type ._llcrypt registered [ 5370.490555] Key type .llcrypt registered [ 5370.527830] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5370.834945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5370.841578] alg: No test for adler32 (adler32-zlib) [ 5371.688374] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 5371.777036] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5373.351160] Key type lgssc registered [ 5373.700765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5376.296408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5376.302978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5377.487329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5378.839790] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5381.707512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5381.856312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5383.871136] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5386.414450] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5387.232786] LustreError: 413019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5387.240247] LustreError: 413019:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5388.256289] LustreError: 413018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5388.262305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5388.264025] LustreError: 413018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5388.750322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5388.881069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5390.274246] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5398.495860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5398.498564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5398.504355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5399.011732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5399.012343] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5399.017116] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5403.519839] Lustre: server umount lustre-OST0000 complete [ 5407.713334] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5407.717133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5407.723153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5407.725423] Lustre: Skipped 1 previous similar message [ 5411.040662] Lustre: server umount lustre-MDT0000 complete [ 5412.410166] LustreError: 413000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782635930 with bad export cookie 1463318758127980948 [ 5412.413344] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5412.414412] LustreError: 413000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5412.571247] Lustre: server umount lustre-MDT0001 complete [ 5416.381899] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5417.443466] Key type lgssc unregistered [ 5417.576685] LNet: 416164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5417.579599] LNetError: 416164:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5417.586381] LNet: Removed LNI 192.168.204.126@tcp [ 5417.918660] Key type .llcrypt unregistered [ 5417.920234] Key type ._llcrypt unregistered [ 5427.340545] Key type ._llcrypt registered [ 5427.342165] Key type .llcrypt registered [ 5427.387089] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5427.805454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5427.849236] alg: No test for adler32 (adler32-zlib) [ 5428.706271] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 5428.796120] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5430.375170] Key type lgssc registered [ 5430.728547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5433.239424] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5433.246668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5448.735488] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5453.855369] LustreError: 417530:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.126@tcp: failed processing log, type 4: rc = -110 [ 5480.479174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5482.373416] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5484.715240] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5487.000475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5487.211395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5487.234688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5488.690285] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5491.762101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5491.926541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5493.479831] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5496.291700] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5498.267326] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5502.431771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5502.434274] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5502.439739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5502.944088] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5502.947175] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5502.952382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5505.505976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5505.508219] Lustre: Skipped 1 previous similar message [ 5506.928598] Lustre: server umount lustre-OST0000 complete [ 5510.624479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5510.625539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5510.628615] Lustre: Skipped 1 previous similar message [ 5510.632150] Lustre: Skipped 1 previous similar message [ 5514.261384] Lustre: server umount lustre-MDT0000 complete [ 5515.586233] LustreError: 417538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782636033 with bad export cookie 7889154162815966243 [ 5515.589532] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5515.590967] LustreError: 417538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5515.741082] Lustre: server umount lustre-MDT0001 complete [ 5519.385972] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5520.466431] Key type lgssc unregistered [ 5520.614511] LNet: 421097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5520.618261] LNetError: 421097:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5520.633493] LNet: Removed LNI 192.168.204.126@tcp [ 5520.999869] Key type .llcrypt unregistered [ 5521.001451] Key type ._llcrypt unregistered [ 5530.278767] Key type ._llcrypt registered [ 5530.279931] Key type .llcrypt registered [ 5530.317504] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5530.719516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5530.726198] alg: No test for adler32 (adler32-zlib) [ 5531.581346] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 5531.669222] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5533.247164] Key type lgssc registered [ 5533.637735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5536.356203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5536.364556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5537.566731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5538.968320] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5541.883365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5542.013904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5543.468643] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5546.336368] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5548.467908] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5551.069628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5551.280280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5553.475221] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5556.096841] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5563.558402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5567.968153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5567.971098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5567.976963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5568.992314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5568.993559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5568.995527] Lustre: Skipped 1 previous similar message [ 5568.999580] Lustre: Skipped 1 previous similar message [ 5571.795554] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 5573.721783] Lustre: server umount lustre-MDT0000 complete [ 5574.113884] LustreError: 422490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5574.121843] LustreError: 422490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5574.566658] LustreError: 422472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.26@tcp arrived at 1782636092 with bad export cookie 2227109147973060836 [ 5574.577819] LustreError: 422472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5575.273370] LustreError: 422471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782636092 with bad export cookie 2227109147973060213 [ 5575.277559] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5575.279651] LustreError: 422471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5578.207760] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5578.211222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5578.218408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5579.231827] LustreError: 422489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5579.237643] LustreError: 422489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5580.255910] LustreError: 422490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5580.262732] LustreError: 422490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5581.429262] Lustre: server umount lustre-OST0000 complete [ 5583.781323] Lustre: server umount lustre-MDT0001 complete [ 5587.802873] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5588.967837] Key type lgssc unregistered [ 5589.115524] LNet: 426191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5589.120113] LNetError: 426191:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5589.128378] LNet: Removed LNI 192.168.204.126@tcp [ 5589.455117] Key type .llcrypt unregistered [ 5589.456643] Key type ._llcrypt unregistered [ 5599.507707] Key type ._llcrypt registered [ 5599.509034] Key type .llcrypt registered [ 5599.551299] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5599.908412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5599.948034] alg: No test for adler32 (adler32-zlib) [ 5600.814680] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 5600.913019] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5602.495116] Key type lgssc registered [ 5602.897824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5605.636876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5605.644238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5606.839805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5608.253205] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5611.243788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5611.384691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5612.901906] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5615.514142] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5617.387816] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5619.736194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5619.900976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5621.955152] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5624.446453] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5632.229798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5637.087796] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5637.090820] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5637.096571] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5637.602400] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5637.607991] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5639.886804] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [ 5641.748133] Lustre: server umount lustre-MDT0001 complete [ 5642.721114] LustreError: 427583:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5642.727460] LustreError: 427583:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5647.840573] LustreError: 427584:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5647.840604] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5647.841517] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5647.842139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5649.599207] Lustre: server umount lustre-OST0000 complete [ 5651.128838] Lustre: server umount lustre-MDT0000 complete [ 5656.045478] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5657.109228] Key type lgssc unregistered [ 5657.258623] LNet: 431287:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5657.261694] LNetError: 431287:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5657.272285] LNet: Removed LNI 192.168.204.126@tcp [ 5657.588050] Key type .llcrypt unregistered [ 5657.589567] Key type ._llcrypt unregistered [ 5666.302946] Key type ._llcrypt registered [ 5666.303950] Key type .llcrypt registered [ 5666.342298] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 5672.709400] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5673.222942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5673.243161] alg: No test for adler32 (adler32-zlib) [ 5674.110321] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 5674.231273] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5675.823163] Key type lgssc registered [ 5676.282449] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5689.547417] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5695.119780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5695.125848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5696.247809] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5696.258192] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5696.292589] Lustre: lustre-MDT0000: new disk, initializing [ 5696.315440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5696.322195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5697.724152] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5703.839953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5703.989197] Lustre: lustre-OST0000: new disk, initializing [ 5703.991874] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5703.992295] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 5703.994262] Lustre: Skipped 1 previous similar message [ 5703.999822] Lustre: 436294:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5704.026315] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5706.151595] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5709.297061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5709.300890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5709.314384] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5710.355098] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5716.122275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5716.169394] Lustre: 436291:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5716.195538] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5716.230438] Lustre: lustre-MDT0001: new disk, initializing [ 5716.272307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5716.287386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 5716.291316] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 5717.681351] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5721.460675] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5721.557631] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5722.967302] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5723.043997] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5727.441867] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [ 5727.712712] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5727.713690] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5727.720696] Lustre: Skipped 1 previous similar message [ 5730.831763] Lustre: server umount lustre-MDT0001 complete [ 5737.952970] LustreError: 436304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5737.961191] LustreError: 436304:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5743.074695] LustreError: 435280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5743.081814] LustreError: 435280:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5745.120103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5745.123473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5745.130369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5745.133041] Lustre: Skipped 2 previous similar messages [ 5748.194155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5748.194319] LustreError: 435285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5749.374100] Lustre: server umount lustre-OST0000 complete [ 5756.832810] Lustre: server umount lustre-MDT0000 complete [ 5761.511066] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5762.524521] Key type lgssc unregistered [ 5762.656352] LNet: 440036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5762.658975] LNetError: 440036:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5762.666332] LNet: Removed LNI 192.168.204.126@tcp [ 5762.977976] Key type .llcrypt unregistered [ 5762.979323] Key type ._llcrypt unregistered [ 5776.116104] Key type ._llcrypt registered [ 5776.117603] Key type .llcrypt registered [ 5776.160577] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5776.266374] Key type .llcrypt unregistered [ 5776.267664] Key type ._llcrypt unregistered [ 5783.326067] Key type ._llcrypt registered [ 5783.327091] Key type .llcrypt registered [ 5783.368283] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5783.703446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5783.749844] alg: No test for adler32 (adler32-zlib) [ 5784.625054] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 5784.719106] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5786.303133] Key type lgssc registered [ 5786.691243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5796.137922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5796.144142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5797.275648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5797.286666] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5797.326221] Lustre: lustre-MDT0000: new disk, initializing [ 5797.351845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5797.360567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5798.793901] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5803.665871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5803.740449] Lustre: 443384:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5803.775686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5803.779103] Lustre: Skipped 1 previous similar message [ 5803.827032] Lustre: lustre-MDT0001: new disk, initializing [ 5803.870872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5803.888224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5803.892139] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5805.974487] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5812.998955] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5817.209647] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5823.320518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5823.564292] Lustre: lustre-OST0000: new disk, initializing [ 5823.567808] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5823.572708] Lustre: 445347:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5823.640619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5828.703631] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5833.822880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5833.835915] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5833.995148] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5837.722834] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5844.451504] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5844.456471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5844.485026] Lustre: Skipped 1 previous similar message [ 5849.583091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5849.595825] Lustre: Skipped 4 previous similar messages [ 5850.591597] Lustre: server umount lustre-MDT0000 complete [ 5854.517039] LustreError: 443376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782636372 with bad export cookie 4499916994893641844 [ 5854.520329] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5854.532768] LustreError: 443376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5854.690488] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5854.701394] Lustre: Skipped 1 previous similar message [ 5854.985650] Lustre: server umount lustre-MDT0001 complete [ 5860.233922] Lustre: server umount lustre-OST0000 complete [ 5866.957849] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 04:46:23 (1782636383) [ 5873.051375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5873.730264] LustreError: 447375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5873.750303] LustreError: 447375:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5873.818559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5879.064464] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5879.274932] LustreError: 447376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5884.388501] LustreError: 447375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5888.604779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5888.990313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5892.895222] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5899.216990] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5903.847732] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5922.986054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5923.181505] Lustre: Found index 0 for lustre-OST0000, updating log [ 5923.319722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5927.070323] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5931.521412] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5945.835564] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 5955.040246] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5955.047253] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5955.072754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5958.600127] Lustre: server umount lustre-OST0000 complete [ 5960.332484] hrtimer: interrupt took 8329032 ns [ 5976.034304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5976.045890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5976.074593] Lustre: Skipped 1 previous similar message [ 5976.092223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5976.108287] Lustre: Skipped 3 previous similar messages [ 5981.686660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5981.697913] Lustre: Skipped 3 previous similar messages [ 5982.753269] LustreError: 448103:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5982.778617] LustreError: 448103:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5982.843781] Lustre: server umount lustre-MDT0000 complete [ 5984.671227] LNet: 1 local NIs in recovery (showing 1): 192.168.204.126@tcp [ 5987.815037] LustreError: 448429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5987.880401] LustreError: 448429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5990.384533] LustreError: 447355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782636508 with bad export cookie 4499916994893643237 [ 5990.387301] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5990.405687] LustreError: 447355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5991.355341] Lustre: server umount lustre-MDT0001 complete [ 6006.010295] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 04:48:41 (1782636521) [ 6620.055085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6620.676468] LustreError: 453814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6620.758881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6625.165994] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6625.767588] LustreError: 453815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6630.882092] LustreError: 453814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6632.577647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6632.876750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6636.778168] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6643.134885] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6647.715202] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6666.544274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6666.869593] Lustre: Found index 0 for lustre-OST0000, updating log [ 6667.085613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6669.123960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 6672.535816] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6674.911146] LNet: 1 local NIs in recovery (showing 1): 192.168.204.126@tcp [ 6678.416518] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6681.323726] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6681.341152] Lustre: Skipped 1 previous similar message [ 6681.348109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6686.631318] Lustre: server umount lustre-OST0000 complete [ 6690.853199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6690.869564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6690.875389] Lustre: Skipped 1 previous similar message [ 6692.900874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6692.917184] Lustre: Skipped 3 previous similar messages [ 6696.186068] Lustre: server umount lustre-MDT0000 complete [ 6696.997212] LustreError: 454872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6697.016957] LustreError: 454872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6699.604371] LustreError: 455682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782637217 with bad export cookie 4499916994893644322 [ 6699.605751] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6699.620897] LustreError: 455682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6700.071357] Lustre: server umount lustre-MDT0001 complete [ 6727.718976] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 6735.973338] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6739.424590] LNet: 1 local NIs in recovery (showing 1): 192.168.204.126@tcp [ 6767.532332] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 05:01:23 (1782637283) [ 6782.985546] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 6785.097666] Key type lgssc unregistered [ 6785.323290] LNet: 460452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6785.329458] LNetError: 460452:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6882.665722] LNet: Removed LNI 192.168.204.126@tcp [ 6883.320267] Key type .llcrypt unregistered [ 6883.322737] Key type ._llcrypt unregistered [ 6895.631801] Key type ._llcrypt registered [ 6895.634477] Key type .llcrypt registered [ 6895.738481] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6896.752730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6896.775714] alg: No test for adler32 (adler32-zlib) [ 6897.774224] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 6898.013794] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 6899.711249] Key type lgssc registered [ 6900.617904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6904.404953] LNet: Added LNI 192.168.204.126@tcp100 [8/256/0/180] [ 6905.308870] LNet: Added LNI 192.168.204.126@tcp101 [8/256/0/180] [ 6906.108882] LNet: Added LNI 192.168.204.126@tcp102 [8/256/0/180] [ 6906.912813] LNet: Added LNI 192.168.204.126@tcp103 [8/256/0/180] [ 6907.617054] LNet: Added LNI 192.168.204.126@tcp104 [8/256/0/180] [ 6908.368677] LNet: Added LNI 192.168.204.126@tcp105 [8/256/0/180] [ 6909.102637] LNet: Added LNI 192.168.204.126@tcp106 [8/256/0/180] [ 6909.922689] LNet: Added LNI 192.168.204.126@tcp107 [8/256/0/180] [ 6910.672465] LNet: Added LNI 192.168.204.126@tcp108 [8/256/0/180] [ 6911.425158] LNet: Added LNI 192.168.204.126@tcp109 [8/256/0/180] [ 6912.076530] LNet: Added LNI 192.168.204.126@tcp110 [8/256/0/180] [ 6912.797489] LNet: Added LNI 192.168.204.126@tcp111 [8/256/0/180] [ 6913.436414] LNet: Added LNI 192.168.204.126@tcp112 [8/256/0/180] [ 6914.052118] LNet: Added LNI 192.168.204.126@tcp113 [8/256/0/180] [ 6914.749856] LNet: Added LNI 192.168.204.126@tcp114 [8/256/0/180] [ 6915.554248] LNet: Added LNI 192.168.204.126@tcp115 [8/256/0/180] [ 6916.246586] LNet: Added LNI 192.168.204.126@tcp116 [8/256/0/180] [ 6916.829086] LNet: Added LNI 192.168.204.126@tcp117 [8/256/0/180] [ 6917.497672] LNet: Added LNI 192.168.204.126@tcp118 [8/256/0/180] [ 6918.286640] LNet: Added LNI 192.168.204.126@tcp119 [8/256/0/180] [ 6918.947305] LNet: Added LNI 192.168.204.126@tcp120 [8/256/0/180] [ 6919.627555] LNet: Added LNI 192.168.204.126@tcp121 [8/256/0/180] [ 6920.253899] LNet: Added LNI 192.168.204.126@tcp122 [8/256/0/180] [ 6920.845726] LNet: Added LNI 192.168.204.126@tcp123 [8/256/0/180] [ 6921.561784] LNet: Added LNI 192.168.204.126@tcp124 [8/256/0/180] [ 6922.237760] LNet: Added LNI 192.168.204.126@tcp125 [8/256/0/180] [ 6922.949104] LNet: Added LNI 192.168.204.126@tcp126 [8/256/0/180] [ 6923.559265] LNet: Added LNI 192.168.204.126@tcp127 [8/256/0/180] [ 6924.215955] LNet: Added LNI 192.168.204.126@tcp128 [8/256/0/180] [ 6924.916482] LNet: Added LNI 192.168.204.126@tcp129 [8/256/0/180] [ 6925.515749] LNet: Added LNI 192.168.204.126@tcp130 [8/256/0/180] [ 6926.173035] LNet: Added LNI 192.168.204.126@tcp131 [8/256/0/180] [ 6926.769878] LNet: Added LNI 192.168.204.126@tcp132 [8/256/0/180] [ 6927.462514] LNet: Added LNI 192.168.204.126@tcp133 [8/256/0/180] [ 6928.141967] LNet: Added LNI 192.168.204.126@tcp134 [8/256/0/180] [ 6928.862727] LNet: Added LNI 192.168.204.126@tcp135 [8/256/0/180] [ 6934.460606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6934.482669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6935.601140] Lustre: 463460:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [ 6935.683143] LNetError: 460920:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [ 6935.702965] LNetError: Protocol error connecting to 192.168.204.126@tcp120 at host 192.168.204.126:988: is it running a compatible version of Lustre? [ 6935.717920] LNetError: Skipped 7 previous similar messages [ 6935.865769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6935.897305] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6935.904210] LNet: 4 local NIs in recovery (showing 4): 192.168.204.126@tcp125, 192.168.204.126@tcp129, 192.168.204.126@tcp120, 192.168.204.126@tcp128 [ 6935.997438] Lustre: lustre-MDT0000: new disk, initializing [ 6936.087629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6936.129439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6936.736089] LNetError: 460918:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [ 6936.756895] LNetError: 460918:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [ 6936.765290] LNetError: Protocol error connecting to 192.168.204.126@tcp103 at host 192.168.204.126:988: is it running a compatible version of Lustre? [ 6936.773228] LNetError: Skipped 1 previous similar message [ 6938.976143] LNetError: 460921:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [ 6938.991517] LNetError: 460921:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [ 6939.001794] LNetError: Protocol error connecting to 192.168.204.126@tcp125 at host 192.168.204.126:988: is it running a compatible version of Lustre? [ 6939.010524] LNetError: Skipped 4 previous similar messages [ 6939.793239] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6948.595318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6948.719807] Lustre: 463477:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6948.785546] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6948.790140] Lustre: Skipped 1 previous similar message [ 6948.855631] Lustre: lustre-MDT0001: new disk, initializing [ 6948.896213] Lustre: 464278:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [ 6948.901041] Lustre: 464278:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6948.986673] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6949.039315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6949.051727] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6950.240843] LNetError: 463486:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [ 6950.263258] LNetError: Protocol error connecting to 192.168.204.126@tcp125 at host 192.168.204.126:988: is it running a compatible version of Lustre? [ 6951.258537] Lustre: 464470:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [ 6951.268014] Lustre: 464470:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6952.548324] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6960.551878] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6965.153279] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6965.601414] LNetError: 463481:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [ 6965.614759] LNetError: Protocol error connecting to 192.168.204.126@tcp119 at host 192.168.204.126:988: is it running a compatible version of Lustre? [ 6970.657720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6970.969435] Lustre: lustre-OST0000: new disk, initializing [ 6970.974600] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6970.989314] Lustre: 465450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6971.081835] Lustre: 465455:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [ 6971.091174] Lustre: 465455:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 6971.138845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6973.795276] LNetError: 463493:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [ 6973.805465] LNetError: 463493:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [ 6973.814136] LNetError: Protocol error connecting to 192.168.204.126@tcp126 at host 192.168.204.126:988: is it running a compatible version of Lustre? [ 6973.822294] LNetError: Skipped 2 previous similar messages [ 6975.621540] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6979.098145] Lustre: 465968:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [ 6979.106193] Lustre: 465968:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6979.160859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6979.171083] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6979.268708] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6982.482942] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6989.792587] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6989.798626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6989.802550] LustreError: Skipped 1 previous similar message [ 6989.821048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6990.179202] LNetError: 463483:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [ 6990.191944] LNetError: 463483:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [ 6990.199273] LNetError: Protocol error connecting to 192.168.204.126@tcp119 at host 192.168.204.126:988: is it running a compatible version of Lustre? [ 6990.208328] LNetError: Skipped 7 previous similar messages [ 6993.712810] Lustre: server umount lustre-OST0000 complete [ 6996.319383] LNet: 1 local NIs in recovery (showing 1): 192.168.204.126@tcp125 [ 7000.033871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7000.036594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7000.037063] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7000.047564] Lustre: Skipped 2 previous similar messages [ 7000.079723] Lustre: Skipped 2 previous similar messages [ 7002.932446] Lustre: server umount lustre-MDT0000 complete [ 7006.044994] LustreError: 463468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782637523 with bad export cookie 2322272366999444138 [ 7006.046352] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7006.055819] LustreError: 463468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7006.444368] Lustre: server umount lustre-MDT0001 complete [ 7015.408958] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7017.547633] Key type lgssc unregistered [ 7017.871182] LNet: 467361:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7017.882070] LNetError: 467361:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7017.887675] LNet: Removed LNI 192.168.204.126@tcp [ 7018.911227] LNet: Removed LNI 192.168.204.126@tcp100 [ 7018.919332] LNetError: 467361:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7018.930809] LNetError: 467361:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7018.936130] LNet: Removed LNI 192.168.204.126@tcp101 [ 7020.000931] LNet: Removed LNI 192.168.204.126@tcp102 [ 7020.007287] LNetError: 467361:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7020.013037] LNetError: 467361:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7020.022625] LNet: Removed LNI 192.168.204.126@tcp103 [ 7021.087840] LNet: Removed LNI 192.168.204.126@tcp104 [ 7022.111398] LNet: Removed LNI 192.168.204.126@tcp105 [ 7022.114702] LNetError: 467361:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7022.119829] LNetError: 467361:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7022.128980] LNet: Removed LNI 192.168.204.126@tcp106 [ 7023.135206] LNet: Removed LNI 192.168.204.126@tcp107 [ 7024.159495] LNet: Removed LNI 192.168.204.126@tcp108 [ 7025.183153] LNet: Removed LNI 192.168.204.126@tcp109 [ 7026.207192] LNet: Removed LNI 192.168.204.126@tcp110 [ 7026.212429] LNetError: 467361:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7026.228529] LNetError: 467361:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7026.234388] LNet: Removed LNI 192.168.204.126@tcp111 [ 7027.295458] LNet: Removed LNI 192.168.204.126@tcp112 [ 7028.319184] LNet: Removed LNI 192.168.204.126@tcp113 [ 7029.343572] LNet: Removed LNI 192.168.204.126@tcp114 [ 7030.367188] LNet: Removed LNI 192.168.204.126@tcp115 [ 7031.391216] LNet: Removed LNI 192.168.204.126@tcp116 [ 7032.415175] LNet: Removed LNI 192.168.204.126@tcp117 [ 7033.439195] LNet: Removed LNI 192.168.204.126@tcp118 [ 7034.463285] LNet: Removed LNI 192.168.204.126@tcp119 [ 7034.469491] LNetError: 467361:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7034.473510] LNetError: 467361:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 7034.478758] LNet: Removed LNI 192.168.204.126@tcp120 [ 7035.487257] LNet: Removed LNI 192.168.204.126@tcp121 [ 7036.511268] LNet: Removed LNI 192.168.204.126@tcp122 [ 7037.535182] LNet: Removed LNI 192.168.204.126@tcp123 [ 7038.559334] LNet: Removed LNI 192.168.204.126@tcp124 [ 7038.572402] LNet: Removed LNI 192.168.204.126@tcp125 [ 7039.583710] LNet: Removed LNI 192.168.204.126@tcp126 [ 7040.607202] LNet: Removed LNI 192.168.204.126@tcp127 [ 7041.631234] LNet: Removed LNI 192.168.204.126@tcp128 [ 7042.656500] LNet: Removed LNI 192.168.204.126@tcp129 [ 7042.665909] LNet: Removed LNI 192.168.204.126@tcp130 [ 7042.677570] LNet: Removed LNI 192.168.204.126@tcp131 [ 7042.679542] LNet: Removed LNI 192.168.204.126@tcp132 [ 7042.685533] LNet: Removed LNI 192.168.204.126@tcp133 [ 7042.691760] LNet: Removed LNI 192.168.204.126@tcp134 [ 7042.723407] LNet: Removed LNI 192.168.204.126@tcp135 [ 7043.437853] Key type .llcrypt unregistered [ 7043.442531] Key type ._llcrypt unregistered [ 7062.464313] Key type ._llcrypt registered [ 7062.466394] Key type .llcrypt registered [ 7062.545458] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 7074.699991] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7075.537816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7075.628059] alg: No test for adler32 (adler32-zlib) [ 7076.620079] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 7076.799470] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7078.440273] Key type lgssc registered [ 7079.288211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7105.220143] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 05:07:01 (1782637621) [ 7117.822545] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7119.635189] Key type lgssc unregistered [ 7119.820788] LNet: 471427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7119.825582] LNetError: 471427:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7119.834578] LNet: Removed LNI 192.168.204.126@tcp [ 7120.393472] Key type .llcrypt unregistered [ 7120.398914] Key type ._llcrypt unregistered [ 7131.557863] Key type ._llcrypt registered [ 7131.559916] Key type .llcrypt registered [ 7131.640700] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7132.181228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7132.195814] alg: No test for adler32 (adler32-zlib) [ 7133.197508] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 7133.427301] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7135.128991] Key type lgssc registered [ 7135.995822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7142.671376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7142.695842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7143.974872] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7143.999693] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7144.082777] Lustre: lustre-MDT0000: new disk, initializing [ 7144.144170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7144.156704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7147.605085] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7155.512414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7155.613451] Lustre: 472616:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7155.670924] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7155.674660] Lustre: Skipped 1 previous similar message [ 7155.737341] Lustre: lustre-MDT0001: new disk, initializing [ 7155.822650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7155.857797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7155.872801] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7159.121245] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7165.779826] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7169.870615] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7174.742834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7175.023352] Lustre: lustre-OST0000: new disk, initializing [ 7175.028532] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7175.036529] Lustre: 474584:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7175.128498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7179.455618] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7182.388426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7182.401904] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7182.496217] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7186.729119] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7189.386905] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing lnet_if_list [ 7193.449264] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7193.470560] LNet: Added LNI 192.168.204.126@tcp10 [8/256/0/180] [ 7197.370222] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7197.390658] LNet: Added LNI 192.168.204.126@tcp11 [8/256/0/180] [ 7199.984654] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7200.003771] LNet: Added LNI 192.168.204.126@tcp12 [8/256/0/180] [ 7202.565381] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7202.582160] LNet: Added LNI 192.168.204.126@tcp13 [8/256/0/180] [ 7204.791221] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7204.813283] LNet: Added LNI 192.168.204.126@tcp14 [8/256/0/180] [ 7206.892906] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7206.913308] LNet: Added LNI 192.168.204.126@tcp15 [8/256/0/180] [ 7209.565222] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7209.586244] LNet: Added LNI 192.168.204.126@tcp16 [8/256/0/180] [ 7212.752882] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7212.782464] LNet: Added LNI 192.168.204.126@tcp17 [8/256/0/180] [ 7216.115988] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7216.137322] LNet: Added LNI 192.168.204.126@tcp18 [8/256/0/180] [ 7219.911626] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7219.948653] LNet: Added LNI 192.168.204.126@tcp19 [8/256/0/180] [ 7223.637237] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7223.663921] LNet: Added LNI 192.168.204.126@tcp20 [8/256/0/180] [ 7227.004428] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7227.026468] LNet: Added LNI 192.168.204.126@tcp21 [8/256/0/180] [ 7230.454907] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7230.488430] LNet: Added LNI 192.168.204.126@tcp22 [8/256/0/180] [ 7234.513706] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7234.548970] LNet: Added LNI 192.168.204.126@tcp23 [8/256/0/180] [ 7238.787974] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7238.824355] LNet: Added LNI 192.168.204.126@tcp24 [8/256/0/180] [ 7242.607415] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7242.628429] LNet: Added LNI 192.168.204.126@tcp25 [8/256/0/180] [ 7246.200418] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7246.222744] LNet: Added LNI 192.168.204.126@tcp26 [8/256/0/180] [ 7249.764540] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7249.787041] LNet: Added LNI 192.168.204.126@tcp27 [8/256/0/180] [ 7253.189344] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7253.206047] LNet: Added LNI 192.168.204.126@tcp28 [8/256/0/180] [ 7256.799678] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7256.827530] LNet: Added LNI 192.168.204.126@tcp29 [8/256/0/180] [ 7261.027467] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7261.062541] LNet: Added LNI 192.168.204.126@tcp30 [8/256/0/180] [ 7264.576200] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7264.592463] LNet: Added LNI 192.168.204.126@tcp31 [8/256/0/180] [ 7268.534619] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7268.553589] LNet: Added LNI 192.168.204.126@tcp32 [8/256/0/180] [ 7272.188185] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7272.205923] LNet: Added LNI 192.168.204.126@tcp33 [8/256/0/180] [ 7275.787796] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7275.806029] LNet: Added LNI 192.168.204.126@tcp34 [8/256/0/180] [ 7279.557569] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7279.580059] LNet: Added LNI 192.168.204.126@tcp35 [8/256/0/180] [ 7283.424809] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7283.444384] LNet: Added LNI 192.168.204.126@tcp36 [8/256/0/180] [ 7287.360889] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7287.380597] LNet: Added LNI 192.168.204.126@tcp37 [8/256/0/180] [ 7291.717362] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7291.740545] LNet: Added LNI 192.168.204.126@tcp38 [8/256/0/180] [ 7296.151863] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7296.172355] LNet: Added LNI 192.168.204.126@tcp39 [8/256/0/180] [ 7300.083715] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7300.106992] LNet: Added LNI 192.168.204.126@tcp40 [8/256/0/180] [ 7304.346369] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7304.365204] LNet: Added LNI 192.168.204.126@tcp41 [8/256/0/180] [ 7309.391385] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7309.414357] LNet: Added LNI 192.168.204.126@tcp42 [8/256/0/180] [ 7314.662924] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7314.690808] LNet: Added LNI 192.168.204.126@tcp43 [8/256/0/180] [ 7320.422760] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7320.446394] LNet: Added LNI 192.168.204.126@tcp44 [8/256/0/180] [ 7325.327257] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7325.346976] LNet: Added LNI 192.168.204.126@tcp45 [8/256/0/180] [ 7329.789844] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7329.808741] LNet: Added LNI 192.168.204.126@tcp46 [8/256/0/180] [ 7335.133109] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7335.166040] LNet: Added LNI 192.168.204.126@tcp47 [8/256/0/180] [ 7341.176572] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7341.196449] LNet: Added LNI 192.168.204.126@tcp48 [8/256/0/180] [ 7346.424108] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7346.444471] LNet: Added LNI 192.168.204.126@tcp49 [8/256/0/180] [ 7352.558777] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7352.602978] LNet: Added LNI 192.168.204.126@tcp50 [8/256/0/180] [ 7367.139113] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7367.144824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7367.155872] Lustre: Skipped 1 previous similar message [ 7367.882918] Lustre: server umount lustre-MDT0000 complete [ 7370.280535] LustreError: 497584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782637887 with bad export cookie 14573849075777095525 [ 7370.283405] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7370.288427] LustreError: 497584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7370.522828] Lustre: server umount lustre-MDT0001 complete [ 7382.771966] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7387.615125] Lustre: 471890:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782637889/real 1782637889] req@000000002641129b x1869231057656704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782637905 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7387.637294] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7387.642519] Lustre: Skipped 1 previous similar message [ 7388.432919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7390.624552] Lustre: 471891:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782637892/real 1782637892] req@00000000a2c94a21 x1869231057656832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782637908 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7392.671151] Lustre: 471892:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782637894/real 1782637894] req@000000000df5f052 x1869231057656960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782637910 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7395.872162] LustreError: 471888:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000006a4b3fa x1869231057657856/t0(0) o250->MGC192.168.204.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7396.107581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7397.156197] Lustre: 471889:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782637898/real 1782637898] req@00000000d74654c6 x1869231057657216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782637914 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7398.965220] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7401.824083] Lustre: 471890:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782637903/real 1782637903] req@0000000083196791 x1869231057657344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782637919 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7403.492335] LustreError: 498784:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7403.508025] LustreError: 498784:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7404.803250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7405.073149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7407.153498] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7408.059386] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7412.839868] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7416.160566] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7430.023723] Lustre: server umount lustre-OST0000 complete [ 7432.672610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7432.682698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7432.684517] Lustre: Skipped 1 previous similar message [ 7432.697081] Lustre: Skipped 3 previous similar messages [ 7437.791977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7437.795502] Lustre: Skipped 1 previous similar message [ 7442.912817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7442.980297] Lustre: server umount lustre-MDT0000 complete [ 7445.271353] LustreError: 472610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782637962 with bad export cookie 14573849075777097877 [ 7445.272348] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7445.279685] LustreError: 472610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7454.908698] Lustre: server umount lustre-MDT0001 complete [ 7461.871256] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7463.681969] Key type lgssc unregistered [ 7463.937706] LNet: 501563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7463.944386] LNetError: 501563:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7463.949140] LNet: Removed LNI 192.168.204.126@tcp [ 7463.951930] LNet: Removed LNI 192.168.204.126@tcp10 [ 7463.954609] LNet: Removed LNI 192.168.204.126@tcp11 [ 7463.957923] LNet: Removed LNI 192.168.204.126@tcp12 [ 7463.960724] LNet: Removed LNI 192.168.204.126@tcp13 [ 7463.966486] LNet: Removed LNI 192.168.204.126@tcp14 [ 7463.968905] LNet: Removed LNI 192.168.204.126@tcp15 [ 7463.972492] LNet: Removed LNI 192.168.204.126@tcp16 [ 7463.974829] LNet: Removed LNI 192.168.204.126@tcp17 [ 7463.977736] LNet: Removed LNI 192.168.204.126@tcp18 [ 7463.980214] LNet: Removed LNI 192.168.204.126@tcp19 [ 7463.982950] LNet: Removed LNI 192.168.204.126@tcp20 [ 7463.989389] LNet: Removed LNI 192.168.204.126@tcp21 [ 7463.991626] LNet: Removed LNI 192.168.204.126@tcp22 [ 7463.994198] LNet: Removed LNI 192.168.204.126@tcp23 [ 7463.996662] LNet: Removed LNI 192.168.204.126@tcp24 [ 7463.999325] LNet: Removed LNI 192.168.204.126@tcp25 [ 7464.001682] LNet: Removed LNI 192.168.204.126@tcp26 [ 7464.003829] LNet: Removed LNI 192.168.204.126@tcp27 [ 7464.006409] LNet: Removed LNI 192.168.204.126@tcp28 [ 7464.009303] LNet: Removed LNI 192.168.204.126@tcp29 [ 7464.011404] LNet: Removed LNI 192.168.204.126@tcp30 [ 7464.013622] LNet: Removed LNI 192.168.204.126@tcp31 [ 7464.015296] LNet: Removed LNI 192.168.204.126@tcp32 [ 7464.017900] LNet: Removed LNI 192.168.204.126@tcp33 [ 7464.020985] LNet: Removed LNI 192.168.204.126@tcp34 [ 7464.023930] LNet: Removed LNI 192.168.204.126@tcp35 [ 7464.028629] LNet: Removed LNI 192.168.204.126@tcp36 [ 7464.032967] LNet: Removed LNI 192.168.204.126@tcp37 [ 7464.036330] LNet: Removed LNI 192.168.204.126@tcp38 [ 7464.039518] LNet: Removed LNI 192.168.204.126@tcp39 [ 7464.044381] LNet: Removed LNI 192.168.204.126@tcp40 [ 7464.046017] LNet: Removed LNI 192.168.204.126@tcp41 [ 7464.051825] LNet: Removed LNI 192.168.204.126@tcp42 [ 7464.054490] LNet: Removed LNI 192.168.204.126@tcp43 [ 7464.062431] LNet: Removed LNI 192.168.204.126@tcp44 [ 7464.065602] LNet: Removed LNI 192.168.204.126@tcp45 [ 7464.068276] LNet: Removed LNI 192.168.204.126@tcp46 [ 7464.070869] LNet: Removed LNI 192.168.204.126@tcp47 [ 7464.072947] LNet: Removed LNI 192.168.204.126@tcp48 [ 7464.075528] LNet: Removed LNI 192.168.204.126@tcp49 [ 7464.084780] LNet: Removed LNI 192.168.204.126@tcp50 [ 7464.671402] Key type .llcrypt unregistered [ 7464.674118] Key type ._llcrypt unregistered [ 7478.109236] Key type ._llcrypt registered [ 7478.111038] Key type .llcrypt registered [ 7478.194817] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 7488.221502] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7488.747781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7488.789069] alg: No test for adler32 (adler32-zlib) [ 7489.777755] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 7489.921940] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7491.535664] Key type lgssc registered [ 7492.254504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7514.254828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7514.281617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7515.497417] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7515.512104] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7515.552640] Lustre: lustre-MDT0000: new disk, initializing [ 7515.606705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7515.619702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7518.131933] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7524.552636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7524.613614] Lustre: 504735:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7524.660035] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7524.665472] Lustre: Skipped 1 previous similar message [ 7524.711835] Lustre: lustre-MDT0001: new disk, initializing [ 7524.766630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7524.796624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7524.805486] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7527.105525] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7532.611834] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7535.627755] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7540.122969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7540.286391] Lustre: *** cfs_fail_loc=90f, val=0*** [ 7540.331656] Lustre: lustre-OST0000: new disk, initializing [ 7540.333806] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7540.337906] Lustre: 506751:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7540.389446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7543.659435] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7546.407780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7546.417281] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7546.503670] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7549.772680] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7556.578100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7556.579649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7556.582322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7556.601729] Lustre: Skipped 1 previous similar message [ 7561.698653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7561.701694] Lustre: Skipped 2 previous similar messages [ 7566.816656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7566.822369] Lustre: Skipped 1 previous similar message [ 7568.351231] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7568.540729] Lustre: server umount lustre-OST0000 complete [ 7570.912322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7570.918072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7570.929062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7571.938114] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7577.056592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7577.061930] Lustre: Skipped 3 previous similar messages [ 7585.247146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7585.361777] Lustre: server umount lustre-MDT0000 complete [ 7587.366285] LustreError: 504727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782638104 with bad export cookie 15327638713727556071 [ 7587.370213] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7587.374458] LustreError: 504727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7587.609713] Lustre: server umount lustre-MDT0001 complete [ 7593.322318] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7595.010865] Key type lgssc unregistered [ 7595.198231] LNet: 508657:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7595.206686] LNetError: 508657:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7595.223746] LNet: Removed LNI 192.168.204.126@tcp [ 7595.754880] Key type .llcrypt unregistered [ 7595.756664] Key type ._llcrypt unregistered [ 7607.426114] Key type ._llcrypt registered [ 7607.427656] Key type .llcrypt registered [ 7607.496554] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7607.928885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7607.946820] alg: No test for adler32 (adler32-zlib) [ 7608.881782] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 7609.023580] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7610.640325] Key type lgssc registered [ 7611.259954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7615.425955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7615.436562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7616.736921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7619.235744] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7624.276610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7624.532559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7627.048809] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7631.069921] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7634.548766] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7638.505210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7638.751804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7642.121698] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7646.498768] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7656.703297] loop0: detected capacity change from 0 to 204800000 [ 7656.797139] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7656.807410] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7656.819210] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7660.014461] loop0: detected capacity change from 0 to 204800000 [ 7660.103651] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7660.110753] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7660.121332] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7662.885619] loop0: detected capacity change from 0 to 204800000 [ 7665.557154] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7665.623296] Lustre: fs15246-OST0000: new disk, initializing [ 7665.625514] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7665.630717] Lustre: 513508:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7665.664527] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7668.553258] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7674.184765] loop1: detected capacity change from 0 to 204800000 [ 7676.970205] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7677.030731] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7677.066776] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7677.122939] Lustre: fs15246-MDT0000: new disk, initializing [ 7677.207210] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7677.221460] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7677.287686] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7678.304850] Lustre: 514629:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000661bd9b4 x1869231555644288/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1782638211 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7679.292854] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7687.394068] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7687.401143] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7687.413607] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7694.308551] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7694.323099] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7697.357384] Lustre: server umount fs15246-MDT0000 complete [ 7699.431724] LustreError: 509914:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7699.443095] LustreError: 509914:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7702.390360] Lustre: server umount fs15246-OST0000 complete [ 7708.646320] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7708.650407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7708.657992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7709.666813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7709.672910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7714.741974] Lustre: server umount lustre-OST0000 complete [ 7716.832106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7716.836690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7716.848513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7716.855254] Lustre: Skipped 1 previous similar message [ 7722.812743] Lustre: server umount lustre-MDT0000 complete [ 7724.661709] LustreError: 509900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782638242 with bad export cookie 2612088615362226571 [ 7724.662971] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7724.670839] LustreError: 509900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7724.897152] Lustre: server umount lustre-MDT0001 complete [ 7730.730917] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7732.500680] Key type lgssc unregistered [ 7732.688587] LNet: 517721:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7732.693816] LNetError: 517721:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7732.707746] LNet: Removed LNI 192.168.204.126@tcp [ 7733.230442] Key type .llcrypt unregistered [ 7733.233331] Key type ._llcrypt unregistered [ 7743.882819] Key type ._llcrypt registered [ 7743.884987] Key type .llcrypt registered [ 7743.939569] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7744.431218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7744.487218] alg: No test for adler32 (adler32-zlib) [ 7745.454792] Lustre: Lustre: Build Version: 2.17.54_83_g562d808 [ 7745.609602] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7747.231235] Key type lgssc registered [ 7747.916645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7768.614900] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 7774.043127] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7791.293080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7791.313068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7792.506840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7792.526888] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7792.576484] Lustre: lustre-MDT0000: new disk, initializing [ 7792.627700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7792.641024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7794.819085] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7800.454416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7800.512155] Lustre: 522161:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7800.551549] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7800.552818] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7800.556917] Lustre: Skipped 1 previous similar message [ 7800.596383] Lustre: lustre-MDT0001: new disk, initializing [ 7800.637077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7800.654463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7800.660489] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7802.698036] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7807.866649] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7810.971915] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7814.663059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7814.854915] Lustre: lustre-OST0000: new disk, initializing [ 7814.858181] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7814.862815] Lustre: 524125:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7814.916763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7818.280483] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7823.292882] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7824.908497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7824.913738] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7824.958913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7825.229224] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7825.328649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7827.091771] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7827.190929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7829.984678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7829.986289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7829.988357] LustreError: Skipped 1 previous similar message [ 7829.989177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7829.994251] Lustre: Skipped 1 previous similar message [ 7834.289706] Lustre: server umount lustre-OST0000 complete [ 7836.641016] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7836.644741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7836.654912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7836.659547] Lustre: Skipped 1 previous similar message [ 7840.226435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7840.226475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7842.307350] Lustre: server umount lustre-MDT0000 complete [ 7844.466170] LustreError: 522151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782638362 with bad export cookie 9084181311513457587 [ 7844.467313] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7844.473122] LustreError: 522151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7844.739496] Lustre: server umount lustre-MDT0001 complete [ 7857.654334] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 7862.575135] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7877.165142] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7759 sec ========== 05:19:53 (1782638393) [ 7877.850494] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:19:54 (1782638394) === [ 7879.398047] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:19:55 (1782638395) === [ 7897.467337] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7898.942824] Key type lgssc unregistered [ 7899.140417] LNet: 530763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7899.150150] LNetError: 530763:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7899.169421] LNet: Removed LNI 192.168.204.126@tcp [ 7899.682691] Key type .llcrypt unregistered [ 7899.684806] Key type ._llcrypt unregistered