[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 488536685 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002390] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.006799] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009140] LSM: Security Framework initializing [ 0.010050] Yama: becoming mindful. [ 0.011032] SELinux: Initializing. [ 0.012074] *** VALIDATE selinux *** [ 0.021677] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026668] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028024] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029126] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030122] *** VALIDATE tmpfs *** [ 0.031499] *** VALIDATE proc *** [ 0.032276] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.034249] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037033] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.041160] debug: unmapping init [mem 0xffffffffb9c59000-0xffffffffb9c60fff] [ 0.044000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044721] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045024] ... version: 2 [ 0.046013] ... bit width: 48 [ 0.047013] ... generic registers: 4 [ 0.048014] ... value mask: 0000ffffffffffff [ 0.049017] ... max period: 00007fffffffffff [ 0.050017] ... fixed-purpose events: 3 [ 0.051015] ... event mask: 000000070000000f [ 0.052306] rcu: Hierarchical SRCU implementation. [ 0.054542] smp: Bringing up secondary CPUs ... [ 0.055579] x86: Booting SMP configuration: [ 0.056027] .... node #0, CPUs: #1 #2 #3 [ 0.062143] smp: Brought up 1 node, 4 CPUs [ 0.064017] smpboot: Max logical packages: 1 [ 0.065014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.274928] node 0 deferred pages initialised in 207ms [ 0.278233] devtmpfs: initialized [ 0.279247] x86/mm: Memory block size: 128MB [ 0.281796] gcov: version magic: 0x41383552 [ 0.283402] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.284086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.285241] pinctrl core: initialized pinctrl subsystem [ 0.286203] [ 0.286676] ************************************************************* [ 0.287018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.288018] ** ** [ 0.289012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.290013] ** ** [ 0.291019] ** This means that this kernel is built to expose internal ** [ 0.292013] ** IOMMU data structures, which may compromise security on ** [ 0.293013] ** your system. ** [ 0.294017] ** ** [ 0.295018] ** If you see this message and you are not debugging the ** [ 0.296019] ** kernel, report this immediately to your vendor! ** [ 0.297018] ** ** [ 0.298017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.299014] ************************************************************* [ 0.300656] NET: Registered protocol family 16 [ 0.301380] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.302068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.303069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.304452] cpuidle: using governor menu [ 0.306822] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.309763] PCI: Using configuration type 1 for base access [ 0.312156] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.321163] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.325052] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.328186] cryptd: max_cpu_qlen set to 1000 [ 0.331326] ACPI: Added _OSI(Module Device) [ 0.332017] ACPI: Added _OSI(Processor Device) [ 0.333018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.334017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.337649] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.341312] ACPI: Interpreter enabled [ 0.342057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.343014] ACPI: Using IOAPIC for interrupt routing [ 0.344096] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.345259] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.354603] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.355043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.356025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.357083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.359319] acpiphp: Slot [2] registered [ 0.360104] acpiphp: Slot [5] registered [ 0.361112] acpiphp: Slot [6] registered [ 0.362140] acpiphp: Slot [7] registered [ 0.363109] acpiphp: Slot [8] registered [ 0.364118] acpiphp: Slot [9] registered [ 0.365123] acpiphp: Slot [10] registered [ 0.366119] acpiphp: Slot [3] registered [ 0.367101] acpiphp: Slot [4] registered [ 0.368103] acpiphp: Slot [11] registered [ 0.369126] acpiphp: Slot [12] registered [ 0.370106] acpiphp: Slot [13] registered [ 0.371126] acpiphp: Slot [14] registered [ 0.372131] acpiphp: Slot [15] registered [ 0.373118] acpiphp: Slot [16] registered [ 0.374078] acpiphp: Slot [17] registered [ 0.375062] acpiphp: Slot [18] registered [ 0.375925] acpiphp: Slot [19] registered [ 0.377136] acpiphp: Slot [20] registered [ 0.378140] acpiphp: Slot [21] registered [ 0.380111] acpiphp: Slot [22] registered [ 0.382161] acpiphp: Slot [23] registered [ 0.383149] acpiphp: Slot [24] registered [ 0.385231] acpiphp: Slot [25] registered [ 0.387177] acpiphp: Slot [26] registered [ 0.388122] acpiphp: Slot [27] registered [ 0.390167] acpiphp: Slot [28] registered [ 0.391125] acpiphp: Slot [29] registered [ 0.393159] acpiphp: Slot [30] registered [ 0.394171] acpiphp: Slot [31] registered [ 0.396142] PCI host bridge to bus 0000:00 [ 0.397031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.400035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.402035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.405036] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.408038] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.410035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.412225] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.416072] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.419065] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.428512] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.433068] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.436023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.440031] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.442027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.445704] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.447736] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.451059] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.453871] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.459022] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.471967] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.478021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.485169] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.491029] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.497021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.515015] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.524577] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.530029] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.537027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.553023] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.562862] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.569019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.576022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.589022] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.601319] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.608021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.614020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.627030] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.636249] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.642015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.648016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.660015] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.670069] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.676017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.683017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.700026] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.711871] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.714455] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.717373] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.719383] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.722248] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.726241] iommu: Default domain type: Passthrough [ 0.727622] SCSI subsystem initialized [ 0.729194] ACPI: bus type USB registered [ 0.730113] usbcore: registered new interface driver usbfs [ 0.732132] usbcore: registered new interface driver hub [ 0.734103] usbcore: registered new device driver usb [ 0.735203] pps_core: LinuxPPS API ver. 1 registered [ 0.737014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.740091] PTP clock support registered [ 0.742192] EDAC MC: Ver: 3.0.0 [ 0.743546] PCI: Using ACPI for IRQ routing [ 0.746694] NetLabel: Initializing [ 0.748015] NetLabel: domain hash size = 128 [ 0.749013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.751129] NetLabel: unlabeled traffic allowed by default [ 0.753213] vgaarb: loaded [ 0.755322] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.756018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.767435] clocksource: Switched to clocksource kvm-clock [ 0.882136] VFS: Disk quotas dquot_6.6.0 [ 0.883947] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.886749] *** VALIDATE ramfs *** [ 0.888019] *** VALIDATE hugetlbfs *** [ 0.889865] pnp: PnP ACPI init [ 0.892520] pnp: PnP ACPI: found 6 devices [ 0.914966] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.918377] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.920898] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.923127] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.925621] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.928207] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.931608] NET: Registered protocol family 2 [ 0.936729] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.941903] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.945878] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.951978] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.955676] TCP: Hash tables configured (established 65536 bind 65536) [ 0.958717] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.961863] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.964651] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.967736] NET: Registered protocol family 1 [ 0.970674] RPC: Registered named UNIX socket transport module. [ 0.972946] RPC: Registered udp transport module. [ 0.974837] RPC: Registered tcp transport module. [ 0.976689] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.979043] NET: Registered protocol family 44 [ 0.980689] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.982748] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.984717] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.987077] PCI: CLS 0 bytes, default 64 [ 0.988557] Unpacking initramfs... [ 2.415799] debug: unmapping init [mem 0xffff9b5c7cc54000-0xffff9b5c7ffbffff] [ 2.419671] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.421577] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.424735] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.928447] Initialise system trusted keyrings [ 2.930241] Key type blacklist registered [ 2.932703] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.942456] zbud: loaded [ 2.945697] *** VALIDATE nfs *** [ 2.947179] *** VALIDATE nfs4 *** [ 2.948742] pstore: using deflate compression [ 2.952032] Platform Keyring initialized [ 3.060662] NET: Registered protocol family 38 [ 3.062786] Key type asymmetric registered [ 3.064378] Asymmetric key parser 'x509' registered [ 3.066334] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.069222] io scheduler mq-deadline registered [ 3.071076] io scheduler kyber registered [ 3.072792] io scheduler bfq registered [ 3.074628] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.077473] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.080567] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.083162] ACPI: Power Button [PWRF] [ 3.174706] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.264790] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.444252] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.537264] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.712704] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.740877] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.769289] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.777623] Non-volatile memory driver v1.3 [ 3.779105] Linux agpgart interface v0.103 [ 3.813330] virtio_blk virtio1: [vda] 133896 512-byte logical blocks (68.6 MB/65.4 MiB) [ 3.815478] vda: detected capacity change from 0 to 68554752 [ 3.830874] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.832892] vdb: detected capacity change from 0 to 1073741824 [ 3.848281] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.850334] vdc: detected capacity change from 0 to 2621440000 [ 3.883775] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.886978] vdd: detected capacity change from 0 to 2621440000 [ 3.908992] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.910902] vde: detected capacity change from 0 to 4294967296 [ 3.928305] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.930733] vdf: detected capacity change from 0 to 4294967296 [ 3.939087] libphy: Fixed MDIO Bus: probed [ 3.944320] usbcore: registered new interface driver usbserial_generic [ 3.946137] usbserial: USB Serial support registered for generic [ 3.947943] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.951111] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.952607] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.954548] mousedev: PS/2 mouse device common for all mice [ 3.957270] rtc_cmos 00:05: RTC can wake from S4 [ 3.959033] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.961201] rtc_cmos 00:05: registered as rtc0 [ 3.964429] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.965163] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.967019] intel_pstate: CPU model not supported [ 3.971716] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.974163] hid: raw HID events driver (C) Jiri Kosina [ 3.975988] usbcore: registered new interface driver usbhid [ 3.977462] usbhid: USB HID core driver [ 3.978587] drop_monitor: Initializing network drop monitor service [ 3.980032] Initializing XFRM netlink socket [ 3.981519] NET: Registered protocol family 10 [ 3.984158] Segment Routing with IPv6 [ 3.985263] NET: Registered protocol family 17 [ 3.986877] mpls_gso: MPLS GSO support [ 3.992770] RAS: Correctable Errors collector initialized. [ 3.994418] AVX version of gcm_enc/dec engaged. [ 3.995642] AES CTR mode by8 optimization enabled [ 4.075836] sched_clock: Marking stable (4075803306, 0)->(5036143670, -960340364) [ 4.078706] registered taskstats version 1 [ 4.080481] Loading compiled-in X.509 certificates [ 4.081770] zswap: loaded using pool lzo/zbud [ 4.105531] Key type big_key registered [ 4.116737] Key type encrypted registered [ 4.118195] ima: No TPM chip found, activating TPM-bypass! [ 4.119354] ima: Allocated hash algorithm: sha1 [ 4.120460] ima: No architecture policies found [ 4.121517] evm: Initialising EVM extended attributes: [ 4.122691] evm: security.selinux [ 4.123578] evm: security.ima [ 4.124343] evm: security.capability [ 4.125066] evm: HMAC attrs: 0x1 [ 4.127077] rtc_cmos 00:05: setting system clock to 2025-10-23 18:43:57 UTC (1761245037) [ 4.132498] debug: unmapping init [mem 0xffffffffbac03000-0xffffffffbadfffff] [ 4.134585] debug: unmapping init [mem 0xffffffffb9982000-0xffffffffb9c58fff] [ 4.142110] Write protecting the kernel read-only data: 28672k [ 4.144717] debug: unmapping init [mem 0xffffffffb8003000-0xffffffffb81fffff] [ 4.146867] debug: unmapping init [mem 0xffffffffb8914000-0xffffffffb89fffff] [ 4.179390] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.184949] systemd[1]: Detected virtualization kvm. [ 4.186469] systemd[1]: Detected architecture x86-64. [ 4.187398] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.211287] systemd[1]: No hostname configured. [ 4.212556] systemd[1]: Set hostname to . [ 4.214252] random: systemd: uninitialized urandom read (16 bytes read) [ 4.216103] systemd[1]: Initializing machine ID from random generator. [ 4.264319] random: ln: uninitialized urandom read (6 bytes read) [ 4.361595] random: systemd: uninitialized urandom read (16 bytes read) [ 4.364046] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.367233] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.371530] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Reached target Slices. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.046461] device-mapper: uevent: version 1.0.3 [ 5.048897] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.823822] virtio_net virtio0 ens2: renamed from eth0 [ 5.871508] random: fast init done [ 5.922959] scsi host0: ata_piix [ 5.939967] scsi host1: ata_piix [ 5.941602] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.943446] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.248428] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.630447] random: crng init done [ 10.631822] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.116460] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.301919] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.561813] SELinux: Disabled at runtime. [ 12.624153] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.632191] systemd[1]: Detected virtualization kvm. [ 12.633919] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.081526] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.084657] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.089658] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.093344] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.095992] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.105512] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.113129] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices.[ 13.138577] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.568204] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.966241] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.018584] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.207754] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.217761] EDAC sbridge: Ver: 1.1.2 [ 15.872960] Key type dns_resolver registered [ 16.181217] NFS: Registering the id_resolver key type [ 16.183728] Key type id_resolver registered [ 16.185352] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg103-server login: [ 64.518535] libcfs: loading out-of-tree module taints kernel. [ 64.540247] Key type ._llcrypt registered [ 64.541975] Key type .llcrypt registered [ 64.657949] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 78.982299] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 80.481845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 80.505689] alg: No test for adler32 (adler32-zlib) [ 81.898280] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 82.608286] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 84.328226] Key type lgssc registered [ 85.477280] Lustre: Echo OBD driver; http://www.lustre.org/ [ 97.087098] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 98.212735] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 104.747592] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 110.603536] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 115.729739] hrtimer: interrupt took 5116356 ns [ 116.630193] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 128.100349] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 138.638398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 138.671365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 138.685162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 139.956992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 139.996666] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 140.089704] Lustre: lustre-MDT0000: new disk, initializing [ 140.167868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 140.192270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 143.347805] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 154.039152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 154.104818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 154.175266] Lustre: 6506:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 154.201645] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 154.207831] Lustre: Skipped 1 previous similar message [ 154.295115] Lustre: lustre-MDT0001: new disk, initializing [ 154.366191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 154.401412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 154.413667] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 157.243439] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 161.235728] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 167.744361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 167.797255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 168.005318] Lustre: lustre-OST0000: new disk, initializing [ 168.010247] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 168.058715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 168.513902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 168.523287] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 168.613095] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 172.258595] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.299609] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 185.387359] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 185.474646] Lustre: lustre-OST0001: new disk, initializing [ 185.477292] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 185.534718] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 192.210970] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.582165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 193.602973] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 193.666147] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 205.600482] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 214.113664] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 224.565564] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing check_logdir /tmp/testlogs/ [ 229.923220] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing yml_node [ 235.289084] Lustre: DEBUG MARKER: Client: 2.16.59.45 [ 237.860451] Lustre: DEBUG MARKER: MDS: 2.16.59.45 [ 240.472985] Lustre: DEBUG MARKER: OSS: 2.16.59.45 [ 241.892881] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Oct 23 14:47:54 EDT 2025 [ 257.386723] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 258.564243] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 275.426060] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 275.426912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 275.435271] Lustre: Skipped 2 previous similar messages [ 279.397138] LustreError: 12218:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 279.489085] Lustre: server umount lustre-MDT0000 complete [ 282.024944] LustreError: 6497:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245315 with bad export cookie 1187753632321789659 [ 282.028230] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 282.037016] LustreError: 6497:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 285.667552] LustreError: 6513:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 285.667654] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 285.667954] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 285.667959] Lustre: Skipped 3 previous similar messages [ 285.674397] LustreError: 6513:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 6 previous similar messages [ 285.695059] Lustre: Skipped 2 previous similar messages [ 287.715169] LustreError: 6511:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 287.715721] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 287.751248] Lustre: Skipped 2 previous similar messages [ 288.267783] LustreError: 12419:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 288.277501] LustreError: 12419:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 288.503433] Lustre: server umount lustre-MDT0001 complete [ 291.812318] LustreError: 12620:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 291.817513] LustreError: 12620:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 291.869275] Lustre: server umount lustre-OST0000 complete [ 294.746258] LustreError: 12821:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 294.749254] LustreError: 12821:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 294.860244] Lustre: server umount lustre-OST0001 complete [ 299.029748] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 307.081713] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 316.947568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 322.801239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 328.820102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 335.336842] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 342.370755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 342.449485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 342.720973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 342.768279] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 342.859388] Lustre: lustre-MDT0000: new disk, initializing [ 343.006303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 343.031688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 346.857897] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 358.121801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 358.205304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 358.364182] Lustre: 15005:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 358.448787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 358.453323] Lustre: Skipped 1 previous similar message [ 358.544852] Lustre: lustre-MDT0001: new disk, initializing [ 358.650754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 358.680811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 358.703384] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 358.883335] Lustre: 15806:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 358.887892] Lustre: 15806:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 358.892756] Lustre: 15806:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 358.912966] Lustre: 15806:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 358.933073] Lustre: 15806:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 358.939302] Lustre: 15806:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 362.607959] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 370.141990] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 373.169125] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 378.569419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 378.626055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 378.805736] Lustre: lustre-OST0000: new disk, initializing [ 378.811687] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 378.903952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 380.599511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 380.603456] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 380.705544] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 383.063943] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 389.640408] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 393.641735] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 393.937992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 397.616309] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 397.874607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 401.377340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 401.385020] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 401.405606] LustreError: Skipped 1 previous similar message [ 401.412397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 401.446090] Lustre: Skipped 1 previous similar message [ 406.086345] LustreError: 17948:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 406.094626] LustreError: 17948:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 406.296605] Lustre: server umount lustre-OST0000 complete [ 411.617530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 411.639374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 411.651335] Lustre: Skipped 2 previous similar messages [ 416.913878] LustreError: 18150:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 416.943505] LustreError: 18150:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 417.080780] Lustre: server umount lustre-MDT0000 complete [ 420.698500] LustreError: 14998:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245454 with bad export cookie 1187753632321791227 [ 420.700859] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 420.724759] LustreError: 14998:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 421.130745] Lustre: server umount lustre-MDT0001 complete [ 424.757331] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 14:50:57 (1761245457) [ 426.538781] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 427.886425] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 14:51:00 (1761245460) [ 433.763940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 434.179738] LustreError: 19020:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 434.196615] LustreError: 19020:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 434.244888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 437.646689] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 439.274062] LustreError: 19020:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 444.387682] LustreError: 19021:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 445.006524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 448.769743] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 453.365155] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 456.456505] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 462.980966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 467.324859] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 471.959856] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 489.957835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 489.967800] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.970306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 490.008403] Lustre: Skipped 2 previous similar messages [ 490.011045] Lustre: Skipped 2 previous similar messages [ 494.461097] LustreError: 21291:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 494.466249] LustreError: 21291:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 494.592862] Lustre: server umount lustre-OST0000 complete [ 500.195966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.216452] Lustre: Skipped 1 previous similar message [ 503.394596] Lustre: server umount lustre-MDT0000 complete [ 505.314043] LustreError: 19020:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 505.326437] LustreError: 19020:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 506.776878] LustreError: 19001:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245540 with bad export cookie 1187753632321792151 [ 506.777485] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 506.798103] LustreError: 19001:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 507.120581] Lustre: server umount lustre-MDT0001 complete [ 518.188131] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 521.426646] Key type lgssc unregistered [ 521.896809] LNet: 22227:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 521.918707] LNetError: 22227:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 521.990274] LNet: Removed LNI 192.168.201.103@tcp [ 523.266465] Key type .llcrypt unregistered [ 523.269435] Key type ._llcrypt unregistered [ 546.997648] Key type ._llcrypt registered [ 546.999304] Key type .llcrypt registered [ 547.121335] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 548.345241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 548.413579] alg: No test for adler32 (adler32-zlib) [ 549.444335] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 549.637899] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 551.296262] Key type lgssc registered [ 552.222706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 558.908734] loop: module loaded [ 558.979973] loop0: detected capacity change from 0 to 204800000 [ 559.137142] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 559.153369] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 559.417848] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 559.460097] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 563.758910] loop0: detected capacity change from 0 to 204800000 [ 563.901463] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 563.909237] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 563.917143] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 564.015886] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 567.540041] loop0: detected capacity change from 0 to 204800000 [ 572.182115] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 572.212525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 572.227207] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 573.481230] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 573.511288] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 573.558654] Lustre: lustre-MDT0000: new disk, initializing [ 573.650210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 573.664116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 573.684717] Lustre: 24363:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 466, rollback = 2 [ 573.691744] Lustre: 24363:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 573.696155] Lustre: 24363:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 2/466/0 [ 573.701348] Lustre: 24363:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 573.708132] Lustre: 24363:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 [ 573.711752] Lustre: 24363:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 577.038986] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 584.601299] loop1: detected capacity change from 0 to 204800000 [ 589.668505] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 589.740582] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 590.227780] Lustre: lustre-OST07c6: new disk, initializing [ 590.233895] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 590.244533] Lustre: Skipped 1 previous similar message [ 590.322362] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 595.591232] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 599.631224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 599.638937] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 599.672574] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 604.742088] Lustre: 25893:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 604.760562] Lustre: 25893:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 2 previous similar messages [ 604.765400] Lustre: 25893:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 604.769875] Lustre: 25893:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 604.775986] Lustre: 25893:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 604.781211] Lustre: 25893:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 604.786028] Lustre: 25893:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 604.789746] Lustre: 25893:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 604.794701] Lustre: 25893:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 604.804373] Lustre: 25893:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 604.814149] Lustre: 25893:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 604.834280] Lustre: 25893:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 2 previous similar messages [ 621.676246] Lustre: 24389:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 497, rollback = 2 [ 621.682940] Lustre: 24389:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 621.686694] Lustre: 24389:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 621.692153] Lustre: 24389:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 621.697255] Lustre: 24389:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 621.700168] Lustre: 24389:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 626.205916] Lustre: 24388:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 497, rollback = 2 [ 626.214340] Lustre: 24388:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 626.218796] Lustre: 24388:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 626.223951] Lustre: 24388:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 626.229753] Lustre: 24388:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 626.243443] Lustre: 24388:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 626.248607] Lustre: 24388:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 626.254084] Lustre: 24388:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 626.257913] Lustre: 24388:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 626.261703] Lustre: 24388:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 626.265770] Lustre: 24388:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 626.272088] Lustre: 24388:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 634.270097] Lustre: 24388:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 497, rollback = 2 [ 634.284830] Lustre: 24388:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 3 previous similar messages [ 634.291760] Lustre: 24388:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 634.297463] Lustre: 24388:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 634.305452] Lustre: 24388:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 634.314265] Lustre: 24388:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 634.318332] Lustre: 24388:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 634.322976] Lustre: 24388:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 634.327581] Lustre: 24388:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 634.331585] Lustre: 24388:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 634.336081] Lustre: 24388:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 634.340437] Lustre: 24388:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 645.270030] Lustre: 24389:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 497, rollback = 2 [ 645.284689] Lustre: 24389:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 5 previous similar messages [ 645.290802] Lustre: 24389:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 645.296213] Lustre: 24389:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 645.314666] Lustre: 24389:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/497/0 [ 645.321575] Lustre: 24389:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 645.335192] Lustre: 24389:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 645.348934] Lustre: 24389:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 645.353961] Lustre: 24389:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 645.367401] Lustre: 24389:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 645.374290] Lustre: 24389:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 645.392108] Lustre: 24389:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 696.804419] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 696.817080] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 696.837592] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 698.302153] LustreError: 26580:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 698.469417] Lustre: server umount lustre-OST07c6 complete [ 706.323619] LustreError: 27081:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 706.332391] LustreError: 27081:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 706.561774] Lustre: server umount lustre-MDT0000 complete [ 719.577324] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 721.827418] Key type lgssc unregistered [ 722.134560] LNet: 27914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.143376] LNetError: 27914:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 722.162489] LNet: Removed LNI 192.168.201.103@tcp [ 722.928519] Key type .llcrypt unregistered [ 722.933042] Key type ._llcrypt unregistered [ 740.685084] Key type ._llcrypt registered [ 740.695869] Key type .llcrypt registered [ 740.793824] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 741.652885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 741.835610] alg: No test for adler32 (adler32-zlib) [ 742.926629] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 743.196167] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 744.816275] Key type lgssc registered [ 746.184667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 753.249467] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 753.266676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 754.676630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 757.911084] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 764.910145] LustreError: 29167:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 764.946207] LustreError: 29167:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 765.492521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 765.891390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 769.157377] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 773.260696] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 776.607878] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 782.649218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 783.022353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 787.119126] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 791.641402] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 800.444935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 806.787762] LustreError: 31388:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 816.105841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 816.107035] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 816.134560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 820.085142] LustreError: 31640:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 820.258962] Lustre: server umount lustre-OST0000 complete [ 826.343923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 826.362887] Lustre: Skipped 1 previous similar message [ 826.366846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 826.384238] Lustre: Skipped 2 previous similar messages [ 829.489193] LustreError: 31842:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 829.492742] LustreError: 31842:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 829.572927] Lustre: server umount lustre-MDT0000 complete [ 831.459026] LustreError: 30784:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 831.482675] LustreError: 30784:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 832.419962] LustreError: 29147:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245865 with bad export cookie 17175628122278659685 [ 832.425967] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 832.442978] LustreError: 29147:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 832.581993] LustreError: 32044:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 832.585138] LustreError: 32044:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 832.782080] Lustre: server umount lustre-MDT0001 complete [ 841.467947] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 844.036769] Key type lgssc unregistered [ 844.381067] LNet: 32575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.392540] LNetError: 32575:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.413902] LNet: Removed LNI 192.168.201.103@tcp [ 845.138432] Key type .llcrypt unregistered [ 845.142099] Key type ._llcrypt unregistered [ 860.984867] Key type ._llcrypt registered [ 860.986996] Key type .llcrypt registered [ 861.116251] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 862.051558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 862.085549] alg: No test for adler32 (adler32-zlib) [ 863.146851] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 863.348174] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 864.977149] Key type lgssc registered [ 866.123692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 873.979118] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 874.018315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 875.708594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 880.001357] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 886.247092] LustreError: 33828:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 886.260349] LustreError: 33828:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 891.369364] LustreError: 33828:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 891.437966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 892.072435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 895.568573] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 900.423268] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 902.998750] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 908.413967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 908.770174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 913.487965] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 917.510385] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 921.086969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 933.345619] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 933.354303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 933.376790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 936.418208] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 936.423834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 936.442247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 937.556230] LustreError: 36099:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 937.716469] Lustre: server umount lustre-OST0000 complete [ 941.540166] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 941.555756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 941.561880] Lustre: Skipped 2 previous similar messages [ 946.660164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 946.663238] Lustre: Skipped 1 previous similar message [ 951.778067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 951.791226] Lustre: Skipped 1 previous similar message [ 955.362701] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 955.467480] LustreError: 36302:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 955.473957] LustreError: 36302:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 955.584902] Lustre: server umount lustre-MDT0000 complete [ 956.907625] LustreError: 34546:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 956.916137] LustreError: 34546:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 959.279433] LustreError: 33807:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761245992 with bad export cookie 17686938148157897315 [ 959.301167] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 959.401255] LustreError: 36504:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 959.406256] LustreError: 36504:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 959.549296] Lustre: server umount lustre-MDT0001 complete [ 967.676935] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 969.618755] Key type lgssc unregistered [ 969.946610] LNet: 37035:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.960048] LNetError: 37035:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.972786] LNet: Removed LNI 192.168.201.103@tcp [ 970.591514] Key type .llcrypt unregistered [ 970.593304] Key type ._llcrypt unregistered [ 990.859358] Key type ._llcrypt registered [ 990.861447] Key type .llcrypt registered [ 990.951631] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 991.852750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 991.926221] alg: No test for adler32 (adler32-zlib) [ 992.899646] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 993.099889] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 994.754778] Key type lgssc registered [ 995.667709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1002.401917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1002.442852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1003.947275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1007.338609] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1014.250037] LustreError: 38433:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1014.268033] LustreError: 38433:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 1014.548145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1014.874514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1018.406690] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1023.960141] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1027.088672] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1034.297182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1034.556822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1041.554571] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1048.365828] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1048.959351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1055.248443] LustreError: 40692:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1055.329446] Lustre: server umount lustre-MDT0000 complete [ 1055.715886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1055.737225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1055.747762] LustreError: 38433:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1059.070825] LustreError: 38412:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246092 with bad export cookie 9570368115410885643 [ 1059.097807] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1059.299612] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1059.307731] Lustre: Skipped 1 previous similar message [ 1059.308364] LustreError: 38428:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1059.311036] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1059.331937] LustreError: 38428:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 1061.166074] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.3@tcp (stopping) [ 1064.417236] LustreError: 38428:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1064.423247] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1064.428575] LustreError: 38428:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 1065.290225] LustreError: 40892:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1065.301767] LustreError: 40892:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1065.461924] Lustre: server umount lustre-MDT0001 complete [ 1094.116960] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1094.148663] LustreError: 41120:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1094.151745] LustreError: 41120:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1094.275750] Lustre: server umount lustre-OST0000 complete [ 1109.186507] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1112.035927] Key type lgssc unregistered [ 1112.421697] LNet: 41957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1112.439841] LNetError: 41957:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1112.468202] LNet: Removed LNI 192.168.201.103@tcp [ 1113.483060] Key type .llcrypt unregistered [ 1113.484882] Key type ._llcrypt unregistered [ 1135.551367] Key type ._llcrypt registered [ 1135.553130] Key type .llcrypt registered [ 1135.669062] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1136.289227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1136.337860] alg: No test for adler32 (adler32-zlib) [ 1137.475936] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 1137.833652] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1139.528700] Key type lgssc registered [ 1141.025410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1149.300371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1149.318922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1150.932126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1154.474307] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1161.189901] LustreError: 43353:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1161.214675] LustreError: 43353:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 1163.429328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1163.921261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1168.498818] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1174.066213] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1177.648616] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1183.356828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1183.582888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1188.883899] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1194.405646] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1195.894755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1206.244892] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1206.246153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1206.259173] Lustre: Skipped 1 previous similar message [ 1206.261454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1206.265330] Lustre: Skipped 1 previous similar message [ 1210.677410] Lustre: lustre-OST0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 1211.722750] LustreError: 45605:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1211.943152] Lustre: server umount lustre-OST0000 complete [ 1230.307369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1230.326865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1230.345971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1230.351461] Lustre: Skipped 2 previous similar messages [ 1231.842513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1235.941388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1235.946041] Lustre: Skipped 3 previous similar messages [ 1236.096463] LustreError: 45990:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1236.105081] LustreError: 45990:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1236.187721] Lustre: server umount lustre-MDT0000 complete [ 1240.189319] LustreError: 43334:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246273 with bad export cookie 2502194519176762384 [ 1240.212361] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1246.388230] LustreError: 46192:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1246.391513] LustreError: 46192:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1246.584334] Lustre: server umount lustre-MDT0001 complete [ 1256.520128] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1258.975781] Key type lgssc unregistered [ 1259.216557] LNet: 46724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1259.226413] LNetError: 46724:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1259.249555] LNet: Removed LNI 192.168.201.103@tcp [ 1259.889139] Key type .llcrypt unregistered [ 1259.890941] Key type ._llcrypt unregistered [ 1279.788824] Key type ._llcrypt registered [ 1279.790548] Key type .llcrypt registered [ 1279.886361] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1280.888399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1280.907807] alg: No test for adler32 (adler32-zlib) [ 1281.946581] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 1282.162929] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1283.842076] Key type lgssc registered [ 1284.919687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1292.492570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1292.517383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1293.920145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1297.209664] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1304.038957] LustreError: 48102:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1304.081560] LustreError: 48102:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 1306.915530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1307.219201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1310.719245] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1315.874674] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1319.084835] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1324.307967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1324.608566] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1329.176163] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1333.771709] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1338.935882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1343.149268] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1346.022904] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1357.529896] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-10-2315h06m30s [ 1358.689170] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1363.240269] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 1363.428092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1363.438415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1364.305811] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 1364.315262] Lustre: Skipped 3 previous similar messages [ 1366.309532] LustreError: 50667:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1366.449242] Lustre: server umount lustre-MDT0000 complete [ 1369.380319] LustreError: 49141:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1369.552689] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1370.618964] LustreError: 48102:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1375.512020] LustreError: 48099:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1375.538723] LustreError: 48099:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 1376.129359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1376.270588] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1376.476216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1376.553247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1377.513592] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1380.025192] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1383.900561] Lustre: DEBUG MARKER: Wait for df (36134) ... [ 1385.577430] Lustre: DEBUG MARKER: done [ 1392.097723] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1392.103545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1392.129566] Lustre: Skipped 2 previous similar messages [ 1392.139122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1392.153621] Lustre: Skipped 3 previous similar messages [ 1393.121671] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1393.139572] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1395.328255] LustreError: 51725:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1395.345362] LustreError: 51725:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1395.504672] Lustre: server umount lustre-OST0000 complete [ 1403.362220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1403.365419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1403.385527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1403.391811] Lustre: Skipped 1 previous similar message [ 1403.399039] Lustre: Skipped 3 previous similar messages [ 1404.838070] LustreError: 51927:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1404.848981] LustreError: 51927:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1404.987669] Lustre: server umount lustre-MDT0000 complete [ 1408.154394] LustreError: 48083:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246441 with bad export cookie 7592202228365900957 [ 1408.155427] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1408.166196] LustreError: 48083:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1408.367944] LustreError: 52128:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1408.379326] LustreError: 52128:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1408.596206] Lustre: server umount lustre-MDT0001 complete [ 1417.454498] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1419.945961] Key type lgssc unregistered [ 1420.306945] LNet: 52660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.320946] LNetError: 52660:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.335884] LNet: Removed LNI 192.168.201.103@tcp [ 1421.059143] Key type .llcrypt unregistered [ 1421.061445] Key type ._llcrypt unregistered [ 1451.663042] Key type ._llcrypt registered [ 1451.664788] Key type .llcrypt registered [ 1451.762697] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 1463.772925] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1464.598502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1464.666020] alg: No test for adler32 (adler32-zlib) [ 1465.624671] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 1465.761363] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1467.416310] Key type lgssc registered [ 1468.373157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1477.300510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1483.493702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1488.775454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1494.096984] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1499.955516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1499.997173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1500.014588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1501.214226] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1501.264827] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1501.315747] Lustre: lustre-MDT0000: new disk, initializing [ 1501.383392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1501.400973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1504.098417] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1512.873449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1512.924542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1513.015996] Lustre: 56590:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1513.065762] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1513.068603] Lustre: Skipped 1 previous similar message [ 1513.119964] Lustre: lustre-MDT0001: new disk, initializing [ 1513.226421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1513.273925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1513.291895] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1513.361047] Lustre: 57385:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 1513.366665] Lustre: 57385:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 1513.372221] Lustre: 57385:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 1513.386469] Lustre: 57385:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 1513.394084] Lustre: 57385:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 1513.405077] Lustre: 57385:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 1516.657127] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1523.506368] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1526.431784] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1532.668337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1532.720597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1532.894535] Lustre: lustre-OST0000: new disk, initializing [ 1532.899274] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1532.980250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1538.737246] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1542.740218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1542.780221] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1542.949489] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1546.350406] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1550.277931] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1550.450746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1553.546907] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1553.800845] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1558.499202] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1558.511717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1558.527076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1558.530908] Lustre: Skipped 1 previous similar message [ 1561.487604] LustreError: 59533:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1561.714561] Lustre: server umount lustre-OST0000 complete [ 1568.737495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1568.742605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1568.753195] Lustre: Skipped 2 previous similar messages [ 1572.139130] LustreError: 59735:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1572.146865] LustreError: 59735:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1572.276803] Lustre: server umount lustre-MDT0000 complete [ 1575.824481] LustreError: 56582:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246609 with bad export cookie 3058379121694131271 [ 1575.830150] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1575.869970] LustreError: 56582:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1576.000749] LustreError: 59937:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1576.011590] LustreError: 59937:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1576.303873] Lustre: server umount lustre-MDT0001 complete [ 1584.991943] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 15:10:17 (1761246617) [ 1590.689542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1591.239885] LustreError: 60608:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1591.261541] LustreError: 60608:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 1591.324390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1594.953761] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1596.386828] LustreError: 60608:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1601.510553] LustreError: 60608:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1602.773765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1603.131577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1606.240636] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1610.160688] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1612.952809] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1618.487539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1618.761524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1624.218334] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1629.210728] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1645.942681] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-10-2315h11m18s [ 1647.247530] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1653.022718] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1655.037412] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1655.076553] LustreError: 60604:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1675.610050] Lustre: lustre-MDT0000: Client 26c607c2-9587-4d54-a646-3e739ac5a521 (at 192.168.201.3@tcp) reconnecting [ 1695.112124] LustreError: 60604:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1697.033473] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1698.912936] Lustre: DEBUG MARKER: done [ 1704.938473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1704.941206] LustreError: Skipped 1 previous similar message [ 1704.949664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1704.979317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1704.981675] Lustre: Skipped 1 previous similar message [ 1708.002510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1708.014884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1708.021038] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1709.326472] LustreError: 63640:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1709.345035] LustreError: 63640:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1709.872899] Lustre: server umount lustre-OST0000 complete [ 1715.681833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1715.696432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1715.718519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1715.731974] Lustre: Skipped 1 previous similar message [ 1720.265576] LustreError: 63843:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1720.273424] LustreError: 63843:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1720.397328] Lustre: server umount lustre-MDT0000 complete [ 1724.900994] LustreError: 61681:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1724.903479] LustreError: 60589:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246758 with bad export cookie 3058379121694132188 [ 1724.908611] LustreError: 61681:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 1724.918406] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1725.347190] Lustre: server umount lustre-MDT0001 complete [ 1735.138156] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1737.533373] Key type lgssc unregistered [ 1737.849517] LNet: 64576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1737.861562] LNetError: 64576:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1737.876609] LNet: Removed LNI 192.168.201.103@tcp [ 1738.616366] Key type .llcrypt unregistered [ 1738.617909] Key type ._llcrypt unregistered [ 1767.525450] Key type ._llcrypt registered [ 1767.528648] Key type .llcrypt registered [ 1767.622102] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 1781.147053] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1782.041456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1782.066776] alg: No test for adler32 (adler32-zlib) [ 1783.179655] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 1783.477182] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1785.217895] Key type lgssc registered [ 1786.742977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1795.354151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1801.020900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1806.124953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1811.389825] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1817.458458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1817.523534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1817.535361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1818.858835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1818.893815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1818.999041] Lustre: lustre-MDT0000: new disk, initializing [ 1819.121436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1819.140638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1822.752901] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1832.595441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1832.675499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1832.831293] Lustre: 68526:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1832.948334] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1832.965108] Lustre: Skipped 1 previous similar message [ 1833.063383] Lustre: lustre-MDT0001: new disk, initializing [ 1833.220722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1833.267172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1833.273183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1833.383187] Lustre: 69328:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 1833.396574] Lustre: 69328:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 1833.401719] Lustre: 69328:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 1833.420885] Lustre: 69328:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 1833.433401] Lustre: 69328:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 1833.444175] Lustre: 69328:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 1836.978800] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1843.871851] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1848.033728] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1854.306210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1854.398874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1854.791347] Lustre: lustre-OST0000: new disk, initializing [ 1854.794648] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1854.890744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1859.950916] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1860.726782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1860.747044] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1860.886725] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1866.527686] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1869.438280] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1869.602712] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1873.006853] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1873.151577] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1876.450158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1876.451397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1876.468052] LustreError: Skipped 1 previous similar message [ 1876.476956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1876.487301] Lustre: Skipped 1 previous similar message [ 1880.495078] LustreError: 71469:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1880.592848] Lustre: server umount lustre-OST0000 complete [ 1884.641568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1884.645565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1884.652489] Lustre: Skipped 1 previous similar message [ 1884.657650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1886.691035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1886.691531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1889.293322] LustreError: 71670:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1889.297176] LustreError: 71670:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1889.349045] Lustre: server umount lustre-MDT0000 complete [ 1891.805961] LustreError: 68517:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761246925 with bad export cookie 11249314353476639204 [ 1891.810053] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1891.817055] LustreError: 68517:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1891.924801] LustreError: 71873:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1891.934629] LustreError: 71873:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1892.107822] Lustre: server umount lustre-MDT0001 complete [ 1898.121312] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 15:15:31 (1761246931) [ 1899.355995] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1900.686174] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 15:15:33 (1761246933) [ 1908.702624] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1911.625868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1911.916620] LustreError: 72921:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1911.936977] LustreError: 72921:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 1911.966149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1912.733375] Lustre: Failing over lustre-MDT0000 [ 1912.737383] LustreError: 72949:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1912.744604] LustreError: 72949:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1912.751178] LustreError: 72949:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 1912.877175] LustreError: 73001:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1912.880595] LustreError: 73001:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1913.038259] Lustre: server umount lustre-MDT0000 complete [ 1917.751870] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 15:15:50 (1761246950) [ 1923.351619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1923.636969] LustreError: 73571:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1923.690774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1928.012318] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1928.678936] LustreError: 73571:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1933.795839] LustreError: 73571:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1935.744541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1935.994850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1939.356207] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1943.506489] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1946.377813] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1951.167129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1955.379467] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1959.288705] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1974.712940] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1979.364058] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1979.374381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1979.386035] Lustre: Skipped 2 previous similar messages [ 1979.397435] Lustre: Skipped 3 previous similar messages [ 1981.422706] LustreError: 75941:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1981.426699] LustreError: 75941:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1981.530845] Lustre: server umount lustre-MDT0000 complete [ 1984.310815] LustreError: 73550:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761247017 with bad export cookie 11249314353476640401 [ 1984.316578] LustreError: 73550:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1984.318300] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1984.482919] LustreError: 73566:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1984.483260] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1984.603974] Lustre: server umount lustre-MDT0001 complete [ 1986.827806] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1990.312767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1996.699172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2009.063714] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x9c1d97cd95ada601 [ 2009.072512] Lustre: MGC192.168.201.103@tcp: Connection restored to 0@lo (at 0@lo) [ 2009.300062] LustreError: 75179:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2009.322167] LustreError: 75179:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 2009.393913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2009.397605] Lustre: Skipped 1 previous similar message [ 2009.476719] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2011.974613] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2018.219412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2021.151786] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2024.783780] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2024.937535] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2027.198254] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2033.638387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2033.638729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2033.641301] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2033.644331] Lustre: Skipped 1 previous similar message [ 2037.786060] LustreError: 78402:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2037.788821] LustreError: 78402:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2037.872491] Lustre: server umount lustre-MDT0000 complete [ 2038.757073] LustreError: 75179:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2038.792756] LustreError: 75179:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 2040.776274] LustreError: 73552:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761247074 with bad export cookie 11249314353476642305 [ 2040.786245] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2040.793657] LustreError: 73552:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2041.087464] Lustre: server umount lustre-MDT0001 complete [ 2046.127441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2052.791377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2060.256245] Lustre: 66739:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761247077/real 1761247077] req@00000000ace1b51f x1846801141430272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761247093 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2060.273482] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2060.280837] Lustre: Skipped 1 previous similar message [ 2062.304373] Lustre: 66738:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761247079/real 1761247079] req@0000000066938748 x1846801141430400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761247095 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2065.376236] Lustre: 66736:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761247082/real 1761247082] req@00000000c5868bd1 x1846801141430528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761247098 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2066.467265] LustreError: 66735:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ace1b51f x1846801141431168/t0(0) o250->MGC192.168.201.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2066.739833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2066.745103] Lustre: Skipped 1 previous similar message [ 2066.803652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2067.752476] Lustre: 66736:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761247084/real 1761247084] req@00000000b68aa8f1 x1846801141430784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761247100 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2068.795961] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2070.137103] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2071.840166] Lustre: 66738:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761247089/real 1761247089] req@000000007ab6c089 x1846801141430912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761247105 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2075.110211] LustreError: 79460:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2075.142302] LustreError: 79460:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 5 previous similar messages [ 2077.559477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2082.483290] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2083.307815] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2087.843357] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2091.177678] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2098.659432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2098.660535] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2098.663137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2098.663147] Lustre: Skipped 1 previous similar message [ 2098.675596] Lustre: Skipped 2 previous similar messages [ 2102.288712] LustreError: 80861:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2102.296477] LustreError: 80861:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2102.379936] Lustre: server umount lustre-MDT0000 complete [ 2106.238847] LustreError: 73552:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761247139 with bad export cookie 11249314353476643670 [ 2106.244713] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2106.251131] LustreError: 73552:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2106.819804] Lustre: server umount lustre-MDT0001 complete [ 2112.940486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2116.114033] Lustre: DEBUG MARKER: files compared the same [ 2128.461135] Lustre: server umount lustre-OST0000 complete [ 2140.668706] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2143.817563] Key type lgssc unregistered [ 2144.167333] LNet: 82496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2144.174479] LNetError: 82496:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2145.229450] LNet: Removed LNI 192.168.201.103@tcp [ 2146.390128] Key type .llcrypt unregistered [ 2146.391788] Key type ._llcrypt unregistered [ 2174.956839] Key type ._llcrypt registered [ 2174.958837] Key type .llcrypt registered [ 2175.045769] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2175.840903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2175.869667] alg: No test for adler32 (adler32-zlib) [ 2176.849855] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 2177.046627] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2178.704736] Key type lgssc registered [ 2179.591845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2184.799817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2184.825962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2186.171441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2189.099489] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug +malloc all [ 2195.736599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2196.039359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2200.008572] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug +malloc all [ 2204.276575] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2206.663209] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2212.144269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2212.388830] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 2212.453248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2216.317713] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug +malloc all [ 2217.452805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2220.624522] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2231.781787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2231.782790] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2231.820803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2231.836767] Lustre: Skipped 1 previous similar message [ 2236.921578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2236.927620] Lustre: Skipped 1 previous similar message [ 2242.016182] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2242.019807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2242.030225] Lustre: Skipped 1 previous similar message [ 2242.052089] LustreError: 86142:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2242.214670] Lustre: server umount lustre-OST0000 complete [ 2247.137543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2247.137932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2247.145518] Lustre: Skipped 2 previous similar messages [ 2247.167357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2251.858976] LustreError: 86345:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2251.867679] LustreError: 86345:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2251.956867] Lustre: server umount lustre-MDT0000 complete [ 2252.256862] LustreError: 83889:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2252.274097] LustreError: 83889:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 2255.028463] LustreError: 83876:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761247288 with bad export cookie 10642726670479648888 [ 2255.030268] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2255.036837] LustreError: 83876:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2255.171316] LustreError: 86546:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2255.173559] LustreError: 86546:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2255.351186] Lustre: server umount lustre-MDT0001 complete [ 2263.648895] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2265.634181] Key type lgssc unregistered [ 2265.844075] LNet: 87079:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2265.853919] LNetError: 87079:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2265.863168] LNet: Removed LNI 192.168.201.103@tcp [ 2266.412106] Key type .llcrypt unregistered [ 2266.414757] Key type ._llcrypt unregistered [ 2285.783236] Key type ._llcrypt registered [ 2285.784844] Key type .llcrypt registered [ 2285.914958] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2286.625207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2286.665835] alg: No test for adler32 (adler32-zlib) [ 2287.811314] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 2288.020898] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2289.658424] Key type lgssc registered [ 2290.748150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2297.074522] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2297.094681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2312.608378] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2317.730284] LustreError: 88427:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.103@tcp: failed processing log, type 4: rc = -110 [ 2344.416888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2348.460991] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2352.286914] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2357.927821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.043745] Lustre: *** cfs_fail_loc=706, val=0*** [ 2358.341216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2358.396231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2361.220870] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2364.905563] LustreError: 89400:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2364.917404] LustreError: 89400:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 5 previous similar messages [ 2367.101476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2367.349097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2370.640454] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2375.547454] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2378.688442] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2382.822911] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2382.824219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2382.838988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2382.848561] Lustre: Skipped 1 previous similar message [ 2386.690171] LustreError: 90753:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2386.950903] Lustre: server umount lustre-OST0000 complete [ 2393.061681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2393.063660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2393.084195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2393.093227] Lustre: Skipped 1 previous similar message [ 2393.095538] Lustre: Skipped 1 previous similar message [ 2396.640261] LustreError: 90954:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2396.661709] LustreError: 90954:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2396.735893] Lustre: server umount lustre-MDT0000 complete [ 2398.184491] LustreError: 89404:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2398.211935] LustreError: 89404:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 2399.886811] LustreError: 88434:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761247433 with bad export cookie 15328938838317892171 [ 2399.895197] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2399.909305] LustreError: 88434:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2400.028202] LustreError: 91155:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2400.031794] LustreError: 91155:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2400.201099] Lustre: server umount lustre-MDT0001 complete [ 2407.726789] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2409.872627] Key type lgssc unregistered [ 2410.086616] LNet: 91689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2410.092835] LNetError: 91689:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2410.101412] LNet: Removed LNI 192.168.201.103@tcp [ 2410.725409] Key type .llcrypt unregistered [ 2410.727460] Key type ._llcrypt unregistered [ 2427.808562] Key type ._llcrypt registered [ 2427.810258] Key type .llcrypt registered [ 2427.912968] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2428.677595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2428.827542] alg: No test for adler32 (adler32-zlib) [ 2429.780291] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 2429.951107] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2431.617155] Key type lgssc registered [ 2432.555680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2439.520557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2439.540576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2443.790980] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2451.405730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2452.143984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2455.582328] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2457.570072] LustreError: 93735:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2457.585368] LustreError: 93735:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 2461.681814] LustreError: 93735:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2461.703723] LustreError: 93735:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 2463.155192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2463.396584] LustreError: 93735:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2463.407876] LustreError: 93735:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 2463.432293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2467.627779] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2468.835300] LustreError: 93739:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2468.844848] LustreError: 93739:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 2473.954601] LustreError: 93739:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2474.849304] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2474.867480] Lustre: MGS: Client 48b0e679-be31-4da8-bfcf-aa649a92f481 (at 0@lo) reconnecting [ 2475.142864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2475.238362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2478.842761] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2492.900031] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2492.901977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2492.904031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2492.906760] Lustre: Skipped 1 previous similar message [ 2496.703419] LustreError: 95701:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2496.827788] Lustre: server umount lustre-OST0000 complete [ 2503.138944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2503.140633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2503.165324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2503.168671] Lustre: Skipped 1 previous similar message [ 2506.323729] LustreError: 95903:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2506.326393] LustreError: 95903:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2506.347763] Lustre: server umount lustre-MDT0000 complete [ 2508.258765] LustreError: 93739:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.271827] LustreError: 93739:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 2509.398252] LustreError: 96107:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2509.401130] LustreError: 96107:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2509.547300] Lustre: server umount lustre-MDT0001 complete [ 2512.343924] LustreError: 96308:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2512.347176] LustreError: 96308:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2512.399695] Lustre: server umount MGS complete [ 2522.191414] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 15:25:54 (1761247554) [ 2548.081348] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 2554.755162] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2561.720279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2567.064849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2571.732081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2577.151019] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2584.131747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2584.198260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2588.643606] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2596.946830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2597.047830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2597.241605] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2597.290983] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2597.349065] Lustre: lustre-MDT0001: new disk, initializing [ 2597.398588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2597.415772] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2597.425159] Lustre: 100822:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2597.429885] LustreError: 100822:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2597.458247] LustreError: 100824:0:(fld_handler.c:251:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2597.468237] LustreError: 100824:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2597.964106] LustreError: 100824:0:(fld_handler.c:251:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2597.973036] LustreError: 100824:0:(fld_handler.c:251:fld_server_lookup()) Skipped 4347 previous similar messages [ 2597.977920] LustreError: 100824:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2597.986014] LustreError: 100824:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) Skipped 4347 previous similar messages [ 2598.961043] LustreError: 100824:0:(fld_handler.c:251:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2598.967017] LustreError: 100824:0:(fld_handler.c:251:fld_server_lookup()) Skipped 8393 previous similar messages [ 2598.981082] LustreError: 100824:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2598.986599] LustreError: 100824:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) Skipped 8454 previous similar messages [ 2600.677972] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2600.968160] LustreError: 100824:0:(fld_handler.c:251:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2600.981157] LustreError: 100824:0:(fld_handler.c:251:fld_server_lookup()) Skipped 17118 previous similar messages [ 2600.987857] LustreError: 100824:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2600.994519] LustreError: 100824:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) Skipped 17057 previous similar messages [ 2604.969100] LustreError: 100824:0:(fld_handler.c:251:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2604.974612] LustreError: 100824:0:(fld_handler.c:251:fld_server_lookup()) Skipped 32410 previous similar messages [ 2604.989691] LustreError: 100824:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2604.994680] LustreError: 100824:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) Skipped 32446 previous similar messages [ 2609.046720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2609.316670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2609.504890] Lustre: lustre-OST0000: new disk, initializing [ 2609.513392] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2609.577179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2612.969047] LustreError: 100824:0:(fld_handler.c:251:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2612.975425] LustreError: 100824:0:(fld_handler.c:251:fld_server_lookup()) Skipped 77155 previous similar messages [ 2612.989095] LustreError: 100824:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2612.996089] LustreError: 100824:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) Skipped 77160 previous similar messages [ 2613.420376] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2616.362739] Lustre: 102014:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@0000000076ad0125 x1846801819804032/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1761247665 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2619.483112] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2625.893732] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2625.909149] Lustre: MGS: Client 1e0906bf-b871-4fc5-89f7-f5a3dd68a048 (at 0@lo) reconnecting [ 2625.951556] Lustre: 100106:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2625.979451] LustreError: 100106:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2626.014552] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2626.116694] Lustre: lustre-MDT0000: new disk, initializing [ 2626.347148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2626.372880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2626.448186] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2627.426177] Lustre: 102535:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@00000000a8889ac5 x1846801819810304/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1761247676 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2628.970080] LustreError: 100824:0:(fld_handler.c:251:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2628.975083] LustreError: 100824:0:(fld_handler.c:251:fld_server_lookup()) Skipped 120505 previous similar messages [ 2628.994937] LustreError: 100824:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2628.998627] LustreError: 100824:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) Skipped 120628 previous similar messages [ 2629.709442] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2632.900607] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2632.916261] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2636.597638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2636.612417] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2636.648195] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2658.794710] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2658.800675] Lustre: Skipped 1 previous similar message [ 2658.807811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2663.562717] LustreError: 103056:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2663.569978] LustreError: 103056:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2663.747522] Lustre: server umount lustre-OST0000 complete [ 2669.030880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2669.031700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2669.035141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2669.035150] Lustre: Skipped 1 previous similar message [ 2669.048588] Lustre: Skipped 2 previous similar messages [ 2673.327388] LustreError: 103260:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2673.353250] LustreError: 103260:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2673.401219] Lustre: server umount lustre-MDT0000 complete [ 2674.150047] LustreError: 102952:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2674.162162] LustreError: 102952:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 2682.737767] Lustre: server umount lustre-MDT0001 complete [ 2685.974651] Lustre: server umount MGS complete [ 2696.100596] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 15:28:48 (1761247728) [ 2711.856892] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2714.302491] Key type lgssc unregistered [ 2714.546389] LNet: 104998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2714.558391] LNetError: 104998:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2714.588105] LNet: Removed LNI 192.168.201.103@tcp [ 2715.357159] Key type .llcrypt unregistered [ 2715.359096] Key type ._llcrypt unregistered [ 2728.993859] Key type ._llcrypt registered [ 2728.997217] Key type .llcrypt registered [ 2729.103308] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2729.643258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2729.668798] alg: No test for adler32 (adler32-zlib) [ 2730.744101] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 2730.902511] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2732.560296] Key type lgssc registered [ 2733.685919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2739.961906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2739.990324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2741.402646] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2745.005424] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2750.948361] LustreError: 106181:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2750.956468] LustreError: 106181:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 2752.199506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2752.483197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2755.223622] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2759.140860] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2762.028669] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2767.885985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2768.141502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2772.751570] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2777.898151] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2786.554826] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2792.929246] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2792.935779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2792.946205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2797.028772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2797.029641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2797.038705] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2797.044110] Lustre: Skipped 1 previous similar message [ 2802.147178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2802.152802] Lustre: Skipped 1 previous similar message [ 2807.264876] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2807.271728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2807.275171] Lustre: Skipped 1 previous similar message [ 2807.310116] LustreError: 108450:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2807.437349] Lustre: server umount lustre-OST0000 complete [ 2812.386498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2812.395712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2812.400379] Lustre: Skipped 1 previous similar message [ 2816.104129] LustreError: 108655:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2816.109904] LustreError: 108655:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2816.159716] Lustre: server umount lustre-MDT0000 complete [ 2817.506417] LustreError: 106905:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2817.516599] LustreError: 106905:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 2818.818306] LustreError: 106161:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761247852 with bad export cookie 15945587968763330623 [ 2818.820340] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2818.826961] LustreError: 106161:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2818.939511] LustreError: 108858:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2818.942149] LustreError: 108858:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2819.086748] Lustre: server umount lustre-MDT0001 complete [ 2826.992148] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2828.959213] Key type lgssc unregistered [ 2829.169658] LNet: 109391:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2829.173712] LNetError: 109391:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2829.182519] LNet: Removed LNI 192.168.201.103@tcp [ 2829.720840] Key type .llcrypt unregistered [ 2829.723241] Key type ._llcrypt unregistered [ 2840.994921] Key type ._llcrypt registered [ 2840.996672] Key type .llcrypt registered [ 2841.081618] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2841.700230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2841.725789] alg: No test for adler32 (adler32-zlib) [ 2842.663870] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 2842.803116] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2844.437517] Key type lgssc registered [ 2845.330713] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2850.018805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2850.024173] LustreError: 110432:0:(tgt_mount.c:2438:server_fill_super()) cfs_race id 716 sleeping [ 2850.026204] Lustre: Skipped 1 previous similar message [ 2850.031498] LustreError: 110429:0:(tgt_mount.c:2438:server_fill_super()) cfs_fail_race id 716 waking [ 2850.037326] LustreError: 110432:0:(tgt_mount.c:2438:server_fill_super()) cfs_fail_race id 716 awake: rc=4992 [ 2850.047470] LustreError: 110432:0:(tgt_mount.c:2450:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2850.054580] LustreError: 110432:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2850.080133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2851.548652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2859.151655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2859.488401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2863.897672] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2869.392557] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing lsmod [ 2872.396540] LustreError: 111609:0:(tgt_mount.c:2438:server_fill_super()) cfs_race id 716 sleeping [ 2872.405217] LustreError: 111612:0:(tgt_mount.c:2438:server_fill_super()) cfs_fail_race id 716 waking [ 2872.416702] LustreError: 111609:0:(tgt_mount.c:2438:server_fill_super()) cfs_fail_race id 716 awake: rc=4990 [ 2872.421856] LustreError: 111609:0:(tgt_mount.c:2450:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2872.436482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2872.442376] LustreError: 111609:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2872.722344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2875.835743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2875.876200] LustreError: 110491:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5 [ 2879.972121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2879.976328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2879.984521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2880.996820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2880.998506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2881.008581] Lustre: Skipped 1 previous similar message [ 2881.012840] Lustre: Skipped 1 previous similar message [ 2884.138545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2884.148589] Lustre: Skipped 3 previous similar messages [ 2889.192716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2889.198678] Lustre: Skipped 2 previous similar messages [ 2890.208448] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2890.270440] LustreError: 111803:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2890.320500] Lustre: server umount lustre-MDT0000 complete [ 2893.340746] LustreError: 110452:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761247926 with bad export cookie 1766851671421711170 [ 2893.345089] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2893.345669] LustreError: 110452:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2894.310459] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2894.310665] LustreError: 110468:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2894.336107] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2894.344121] LustreError: 110468:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 2894.432775] LustreError: 110905:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006bde177d x1846802252848512/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2894.453179] LustreError: 112008:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2894.455986] LustreError: 112008:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2894.579497] Lustre: server umount lustre-MDT0001 complete [ 2899.657268] LustreError: 112211:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2899.665644] LustreError: 112211:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2899.845405] Lustre: server umount lustre-OST0000 complete [ 2906.679756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2907.035361] LustreError: 112688:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2907.120684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2910.010977] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2912.227887] LustreError: 112688:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2916.028080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2916.271450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2919.529617] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2924.191799] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2927.520948] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2933.513174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2933.887739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2939.035512] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2943.397685] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2953.334962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2958.821502] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2958.830470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2958.842517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2958.860267] Lustre: Skipped 1 previous similar message [ 2959.878129] LustreError: 114955:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2959.883111] LustreError: 114955:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2960.091638] Lustre: server umount lustre-OST0000 complete [ 2963.943530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2963.965186] Lustre: Skipped 1 previous similar message [ 2969.841211] LustreError: 115157:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2969.858410] LustreError: 115157:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2969.946424] Lustre: server umount lustre-MDT0000 complete [ 2973.552343] LustreError: 112669:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761248006 with bad export cookie 1766851671421712115 [ 2973.560174] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2974.037083] Lustre: server umount lustre-MDT0001 complete [ 2982.380636] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2984.397959] Key type lgssc unregistered [ 2984.607730] LNet: 115889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2984.613626] LNetError: 115889:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2984.627738] LNet: Removed LNI 192.168.201.103@tcp [ 2985.239267] Key type .llcrypt unregistered [ 2985.241163] Key type ._llcrypt unregistered [ 3000.843227] Key type ._llcrypt registered [ 3000.844777] Key type .llcrypt registered [ 3000.929103] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3001.578932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3001.593323] alg: No test for adler32 (adler32-zlib) [ 3002.525551] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 3002.715489] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3004.360471] Key type lgssc registered [ 3005.369269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3011.387089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3011.405837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3012.737290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3015.312249] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3022.136484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3022.444169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3025.556565] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3029.758800] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3032.540940] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3037.332523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3037.570538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3038.582761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3041.760949] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3046.426090] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3050.778039] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3053.692062] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3057.121247] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3057.127812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3057.138601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3057.153148] Lustre: Skipped 1 previous similar message [ 3057.168919] Lustre: Skipped 1 previous similar message [ 3058.759124] LustreError: 119620:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3058.915523] Lustre: server umount lustre-OST0000 complete [ 3062.241475] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3062.244095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3062.254225] Lustre: Skipped 1 previous similar message [ 3067.361529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3067.369594] Lustre: Skipped 2 previous similar messages [ 3067.980848] LustreError: 119825:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3067.983977] LustreError: 119825:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3068.120819] Lustre: server umount lustre-MDT0000 complete [ 3071.114881] LustreError: 117246:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761248104 with bad export cookie 13542443220847343349 [ 3071.123837] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3071.262709] LustreError: 120026:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3071.270714] LustreError: 120026:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3071.468731] Lustre: server umount lustre-MDT0001 complete [ 3080.041677] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3082.216748] Key type lgssc unregistered [ 3082.490430] LNet: 120559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3082.496577] LNetError: 120559:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3082.511633] LNet: Removed LNI 192.168.201.103@tcp [ 3083.218684] Key type .llcrypt unregistered [ 3083.226220] Key type ._llcrypt unregistered [ 3097.705338] Key type ._llcrypt registered [ 3097.710289] Key type .llcrypt registered [ 3097.835134] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3099.031707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3099.119540] alg: No test for adler32 (adler32-zlib) [ 3100.244181] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 3100.482580] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3102.168222] Key type lgssc registered [ 3103.309597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3110.886703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3110.925647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3112.404783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3116.110675] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3121.636542] LustreError: 121761:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3121.652830] LustreError: 121761:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 3123.502692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3123.896615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3127.321727] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3131.982790] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3134.907474] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3140.295610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3140.661284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3145.349461] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3149.671525] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3157.043593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3167.204510] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3167.209549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3167.233240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3167.250504] Lustre: Skipped 1 previous similar message [ 3172.325798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3172.328695] Lustre: Skipped 1 previous similar message [ 3172.611188] LustreError: 124030:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3172.821304] Lustre: server umount lustre-OST0000 complete [ 3177.443404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3177.446928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3177.461679] Lustre: Skipped 1 previous similar message [ 3177.468129] Lustre: Skipped 1 previous similar message [ 3182.564273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3182.569802] Lustre: Skipped 1 previous similar message [ 3182.776562] LustreError: 124232:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3182.779198] LustreError: 124232:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3182.872306] Lustre: server umount lustre-MDT0000 complete [ 3186.330183] LustreError: 121740:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761248219 with bad export cookie 9293094441188132259 [ 3186.333857] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3186.351079] LustreError: 121740:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3186.463394] LustreError: 124433:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3186.470857] LustreError: 124433:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3186.681801] Lustre: server umount lustre-MDT0001 complete [ 3196.046850] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3198.227194] Key type lgssc unregistered [ 3198.527917] LNet: 124965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3198.536762] LNetError: 124965:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3198.556601] LNet: Removed LNI 192.168.201.103@tcp [ 3199.308188] Key type .llcrypt unregistered [ 3199.310642] Key type ._llcrypt unregistered [ 3217.918989] Key type ._llcrypt registered [ 3217.920768] Key type .llcrypt registered [ 3218.012912] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3218.734438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3218.807068] alg: No test for adler32 (adler32-zlib) [ 3219.796182] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 3220.057664] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3221.744197] Key type lgssc registered [ 3222.631128] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3229.417566] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3229.445539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3231.091833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3234.918428] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3241.453124] LustreError: 126341:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3241.465446] LustreError: 126341:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 3244.072842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3244.328723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3248.362468] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3253.114567] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3256.283148] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3261.027959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3261.253923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3265.492656] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3269.101366] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3276.594098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3283.849777] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3286.215778] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3286.220924] Lustre: Skipped 2 previous similar messages [ 3289.597235] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3289.600835] Lustre: Skipped 2 previous similar messages [ 3292.227865] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3292.252254] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3297.147940] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.3@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3297.153887] Lustre: Skipped 2 previous similar messages [ 3300.792247] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.3@tcp 192.168.0.[2,10]@tcp [ 3304.526231] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.3/24@tcp in log lustre-MDT0000 [ 3304.530569] Lustre: Skipped 2 previous similar messages [ 3311.187310] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.3/24@tcp [ 3311.189797] Lustre: Skipped 1 previous similar message [ 3317.736790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3317.738138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3317.742432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3317.757601] LustreError: Skipped 2 previous similar messages [ 3317.764379] Lustre: Skipped 1 previous similar message [ 3317.822602] LustreError: 130016:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3317.953622] Lustre: server umount lustre-OST0000 complete [ 3321.313840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3321.317488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3321.324425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3321.327297] Lustre: Skipped 1 previous similar message [ 3322.851230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3322.851404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3327.969584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3327.975265] Lustre: Skipped 2 previous similar messages [ 3333.092474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3333.095212] Lustre: Skipped 1 previous similar message [ 3335.136907] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3335.218368] LustreError: 130217:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3335.223392] LustreError: 130217:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3335.335922] Lustre: server umount lustre-MDT0000 complete [ 3338.209656] LustreError: 127360:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3338.224537] LustreError: 127360:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 3338.722092] LustreError: 126323:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761248372 with bad export cookie 18140785262205462904 [ 3338.724721] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3338.747285] LustreError: 126323:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3338.908691] LustreError: 130419:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3338.911844] LustreError: 130419:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3339.101130] Lustre: server umount lustre-MDT0001 complete [ 3348.185081] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3350.636178] Key type lgssc unregistered [ 3350.885233] LNet: 130951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3350.898742] LNetError: 130951:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3351.916740] LNet: Removed LNI 192.168.201.103@tcp [ 3352.573476] Key type .llcrypt unregistered [ 3352.575268] Key type ._llcrypt unregistered [ 3376.016879] Key type ._llcrypt registered [ 3376.018738] Key type .llcrypt registered [ 3376.121105] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3376.895865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3376.936585] alg: No test for adler32 (adler32-zlib) [ 3378.000917] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 3378.203510] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3379.872510] Key type lgssc registered [ 3381.100287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3388.588397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3388.621865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3390.095231] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3390.130315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3390.135235] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.3/24@tcp [ 3393.811429] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3395.556943] LustreError: 132495:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3395.564197] LustreError: 132495:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 3399.725500] LustreError: 132495:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3402.441493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3402.759348] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3402.777578] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.3/24@tcp [ 3402.817530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3406.697965] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3411.779520] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3414.471346] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3420.891144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3421.291313] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3426.860190] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3431.984309] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3435.639193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3446.244803] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3446.248698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3446.267684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3446.277073] Lustre: Skipped 1 previous similar message [ 3451.361766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3451.458686] LustreError: 134800:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3451.587550] Lustre: server umount lustre-OST0000 complete [ 3456.482594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3456.490527] Lustre: Skipped 1 previous similar message [ 3456.496040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3456.499072] Lustre: Skipped 1 previous similar message [ 3460.323107] LustreError: 135003:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3460.334889] LustreError: 135003:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3460.454656] Lustre: server umount lustre-MDT0000 complete [ 3461.601170] LustreError: 132490:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3461.615566] LustreError: 132490:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 3463.405711] LustreError: 134122:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761248496 with bad export cookie 11194364879136789815 [ 3463.410666] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3463.423969] LustreError: 134122:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3463.559492] LustreError: 135205:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3463.569552] LustreError: 135205:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3463.795646] Lustre: server umount lustre-MDT0001 complete [ 3473.730520] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3476.256867] Key type lgssc unregistered [ 3476.544646] LNet: 135737:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3476.556817] LNetError: 135737:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3476.578629] LNet: Removed LNI 192.168.201.103@tcp [ 3477.384918] Key type .llcrypt unregistered [ 3477.389255] Key type ._llcrypt unregistered [ 3498.367466] Key type ._llcrypt registered [ 3498.369201] Key type .llcrypt registered [ 3498.479477] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 3508.646791] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3509.155552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3509.186812] alg: No test for adler32 (adler32-zlib) [ 3510.290065] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 3510.488718] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3512.144143] Key type lgssc registered [ 3513.095414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3520.117227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3525.726224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3531.817269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3538.774130] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3544.881111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3544.931666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3544.942943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3546.187613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3546.232511] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3546.311807] Lustre: lustre-MDT0000: new disk, initializing [ 3546.380432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3546.393936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3549.836614] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3557.990793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3558.063117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3558.124328] Lustre: 139091:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3558.186173] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3558.189346] Lustre: Skipped 1 previous similar message [ 3558.246256] Lustre: lustre-MDT0001: new disk, initializing [ 3558.318282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3558.356736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3558.362172] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3558.466743] Lustre: 139888:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 3558.476060] Lustre: 139888:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 3558.482173] Lustre: 139888:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 3558.502298] Lustre: 139888:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 3558.509502] Lustre: 139888:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 3558.519613] Lustre: 139888:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 3561.210620] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3567.553052] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3570.785339] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3575.853116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3575.917954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3576.101683] Lustre: lustre-OST0000: new disk, initializing [ 3576.116138] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3576.222362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3580.508292] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3585.104498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3585.117424] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3585.215851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3587.013794] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3589.947388] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3590.118754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3593.116442] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3593.278150] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3595.747939] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3595.752526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3595.760331] LustreError: Skipped 1 previous similar message [ 3595.768681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3595.772576] Lustre: Skipped 1 previous similar message [ 3600.875428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3605.991635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3605.995497] Lustre: Skipped 2 previous similar messages [ 3609.057199] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3609.089251] LustreError: 142033:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3609.214935] Lustre: server umount lustre-OST0000 complete [ 3614.690306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3614.693927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3614.713144] Lustre: Skipped 1 previous similar message [ 3614.719736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3616.226152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3618.351121] LustreError: 142235:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3618.360256] LustreError: 142235:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3618.463514] Lustre: server umount lustre-MDT0000 complete [ 3621.776454] LustreError: 140839:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761248655 with bad export cookie 3948774020734056538 [ 3621.779163] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3621.799280] LustreError: 140839:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3621.942224] LustreError: 142437:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3621.959443] LustreError: 142437:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3622.210281] Lustre: server umount lustre-MDT0001 complete [ 3629.664158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3630.096767] LustreError: 142914:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3630.128445] LustreError: 142914:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 3630.187206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3633.417144] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3635.681679] LustreError: 142914:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3639.556862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3639.832436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3644.266212] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3649.622990] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3653.317947] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3659.633582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3659.901199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3666.537596] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3672.087845] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3675.935662] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3677.112449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3680.584436] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3680.845329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3689.940171] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3690.016804] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3690.176673] Lustre: lustre-OST0001: new disk, initializing [ 3690.189152] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3690.285947] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3695.078677] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3696.236524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3696.247595] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3696.388247] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3701.966553] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3702.163847] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3705.627409] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3705.806413] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3709.047083] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff994ac4cf2800.ost_server_uuid 50 [ 3710.396680] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff994ac4cf2800.ost_server_uuid in FULL state after 0 sec [ 3717.094835] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3717.096670] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3717.099054] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3717.118341] Lustre: Skipped 1 previous similar message [ 3717.122341] Lustre: Skipped 2 previous similar messages [ 3722.365092] LustreError: 147174:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3722.373495] LustreError: 147174:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3722.433064] Lustre: server umount lustre-OST0001 complete [ 3726.817168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3726.826744] LustreError: Skipped 1 previous similar message [ 3726.830368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3726.850210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3726.857335] Lustre: Skipped 3 previous similar messages [ 3727.335155] LustreError: 144513:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3727.350841] LustreError: 144513:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 3731.658190] LustreError: 147376:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3731.672908] LustreError: 147376:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3731.857969] Lustre: server umount lustre-OST0000 complete [ 3737.057904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3737.066648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3737.080121] Lustre: Skipped 1 previous similar message [ 3741.292470] LustreError: 147578:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3741.301184] LustreError: 147578:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3741.388277] Lustre: server umount lustre-MDT0000 complete [ 3742.690536] LustreError: 142910:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3742.705757] LustreError: 142910:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 3744.439601] LustreError: 142895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761248777 with bad export cookie 3948774020734057462 [ 3744.444863] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3744.454831] LustreError: 142895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3744.845440] Lustre: server umount lustre-MDT0001 complete [ 3760.790730] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3762.879379] Key type lgssc unregistered [ 3763.156541] LNet: 148769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3763.162793] LNetError: 148769:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3763.175112] LNet: Removed LNI 192.168.201.103@tcp [ 3763.794625] Key type .llcrypt unregistered [ 3763.796266] Key type ._llcrypt unregistered [ 3792.701667] Key type ._llcrypt registered [ 3792.709551] Key type .llcrypt registered [ 3792.830761] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 3807.716739] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3808.734237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3808.753046] alg: No test for adler32 (adler32-zlib) [ 3809.800989] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 3810.063661] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3811.768285] Key type lgssc registered [ 3812.944725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3821.022459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3826.467107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3831.779616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3837.319413] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3843.537119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3843.601748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3843.621033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3844.913377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3844.970313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3845.044772] Lustre: lustre-MDT0000: new disk, initializing [ 3845.115927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3845.143947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3848.687429] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3858.694607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3858.770805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3858.853121] Lustre: 152743:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3858.921826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3858.924253] Lustre: Skipped 1 previous similar message [ 3859.028216] Lustre: lustre-MDT0001: new disk, initializing [ 3859.194442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3859.276145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3859.298117] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3859.426540] Lustre: 153541:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 3859.431034] Lustre: 153541:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 3859.433854] Lustre: 153541:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 3859.454171] Lustre: 153541:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 3859.467899] Lustre: 153541:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 3859.477865] Lustre: 153541:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 3862.977699] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3869.996666] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3873.377358] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3878.762660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3878.858362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3879.065589] Lustre: lustre-OST0000: new disk, initializing [ 3879.068689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3879.178989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3883.696498] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3889.250654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3889.261387] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3889.371455] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3890.070072] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3893.512509] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3893.742259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3897.845212] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3898.197135] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3904.993673] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3904.996663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3905.034585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3906.315274] LustreError: 155685:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3906.489840] Lustre: server umount lustre-OST0000 complete [ 3910.625354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3910.629085] LustreError: Skipped 1 previous similar message [ 3910.660045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3910.666662] Lustre: Skipped 1 previous similar message [ 3910.704593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3910.707539] Lustre: Skipped 1 previous similar message [ 3913.189170] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3913.192519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3916.214569] LustreError: 155886:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3916.226531] LustreError: 155886:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3916.294649] Lustre: server umount lustre-MDT0000 complete [ 3919.975661] LustreError: 153534:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761248953 with bad export cookie 13047604360963302531 [ 3919.981019] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3919.994644] LustreError: 153534:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3920.274487] LustreError: 156087:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3920.283092] LustreError: 156087:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3920.486759] Lustre: server umount lustre-MDT0001 complete [ 3929.405934] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 15:49:22 (1761248962) [ 3944.912704] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 3954.416690] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3965.966441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3973.219937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3979.410728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3985.873490] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3992.090716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3992.148720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3992.425619] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3992.471517] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3992.607512] Lustre: lustre-MDT0000: new disk, initializing [ 3992.717754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3992.743724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3996.532166] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4006.554803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4006.613534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4006.677726] Lustre: 159081:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4006.732271] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4006.734978] Lustre: Skipped 1 previous similar message [ 4006.782764] Lustre: lustre-MDT0001: new disk, initializing [ 4006.861503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4006.896695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4006.906469] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4006.962761] Lustre: 159878:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 4006.974984] Lustre: 159878:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 4006.979433] Lustre: 159878:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 4006.990343] Lustre: 159878:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4007.004023] Lustre: 159878:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 4007.025180] Lustre: 159878:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4007.052213] Lustre: 159878:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 4007.059134] Lustre: 159878:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4007.070653] Lustre: 159878:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 4007.076034] Lustre: 159878:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4007.079870] Lustre: 159878:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 4007.083657] Lustre: 159878:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4011.032607] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4019.392488] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4024.289586] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4032.893330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4032.975314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4033.342581] Lustre: lustre-OST0000: new disk, initializing [ 4033.361788] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4033.458282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4039.439366] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4043.374974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4043.380198] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4043.482476] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4046.102477] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4049.636734] Lustre: Failing over lustre-OST0000 [ 4049.707141] LustreError: 161522:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4049.714331] LustreError: 161522:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4049.873780] Lustre: server umount lustre-OST0000 complete [ 4067.612448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4067.820734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4067.858554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4067.876502] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4068.155506] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4068.899131] Lustre: 150954:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761249085/real 1761249085] req@0000000071154fc0 x1846803266872960/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1761249101 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4069.924198] Lustre: 150955:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761249087/real 1761249087] req@0000000023a694a4 x1846803266873216/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761249103 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4072.817263] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4072.981340] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [ 4072.983427] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4074.976221] Lustre: 150953:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761249092/real 1761249092] req@00000000f35f0552 x1846803266875008/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761249108 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4075.011550] Lustre: 150953:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4076.636232] Lustre: Failing over lustre-MDT0000 [ 4076.790663] LustreError: 162505:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4076.799777] LustreError: 162505:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4076.944204] Lustre: server umount lustre-MDT0000 complete [ 4078.057641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4078.063716] LustreError: 159880:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4078.067105] Lustre: Skipped 2 previous similar messages [ 4078.072321] LustreError: 159880:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 4080.096160] Lustre: 150955:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761249097/real 1761249097] req@00000000cccaaa0f x1846803266876160/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761249113 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4080.122110] Lustre: 150955:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4083.175258] LustreError: 159880:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4083.206201] LustreError: 159880:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 5 previous similar messages [ 4088.290947] LustreError: 159880:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4088.302600] LustreError: 159880:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 4093.414528] LustreError: 161334:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4093.480383] LustreError: 161334:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 4094.434829] Lustre: 150953:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761249111/real 1761249111] req@00000000ae02c915 x1846803266884608/t0(0) o400->MGC192.168.201.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761249127 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4094.492297] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4095.172492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4098.841251] LustreError: 159087:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4098.857127] LustreError: 159087:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 4 previous similar messages [ 4103.973344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4104.046667] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4104.071222] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4107.923914] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4109.312455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4109.314980] Lustre: Skipped 1 previous similar message [ 4109.338976] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4109.391661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4114.914169] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4114.922984] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4114.931856] Lustre: Skipped 1 previous similar message [ 4114.938566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4114.944792] Lustre: Skipped 1 previous similar message [ 4116.546335] LustreError: 163407:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4116.551297] LustreError: 163407:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4116.775782] Lustre: server umount lustre-OST0000 complete [ 4124.644575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4124.658461] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4134.368129] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4134.486262] LustreError: 163609:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4134.491173] LustreError: 163609:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4134.604145] Lustre: server umount lustre-MDT0000 complete [ 4134.881267] Lustre: 150955:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761249152/real 1761249152] req@00000000dd918173 x1846803266901888/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761249168 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4134.881582] LustreError: 159880:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4134.902788] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4134.916243] LustreError: 159880:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 7 previous similar messages [ 4134.925593] Lustre: Skipped 2 previous similar messages [ 4138.437371] LustreError: 163610:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761249171 with bad export cookie 13047604360963304729 [ 4138.440077] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4138.443737] LustreError: 163610:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4138.786219] Lustre: server umount lustre-MDT0001 complete [ 4148.776896] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 4151.402803] Key type lgssc unregistered [ 4151.743558] LNet: 164345:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4151.760415] LNetError: 164345:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4151.776509] LNet: Removed LNI 192.168.201.103@tcp [ 4152.918047] Key type .llcrypt unregistered [ 4152.919412] Key type ._llcrypt unregistered [ 4190.114789] Key type ._llcrypt registered [ 4190.116932] Key type .llcrypt registered [ 4190.210700] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4191.075812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4191.243174] alg: No test for adler32 (adler32-zlib) [ 4192.275434] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 4192.457320] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 4194.088192] Key type lgssc registered [ 4195.181458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4202.144264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4202.179654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4203.670514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4206.824837] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4213.841799] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4214.241668] LustreError: 165848:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4214.258852] LustreError: 165848:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 4217.587288] LustreError: 165877:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4217.593871] LustreError: 165877:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4217.599891] LustreError: 165877:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 13, retries 0, failed: rc = -5 [ 4217.688530] LustreError: 166401:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4217.931758] Lustre: server umount lustre-MDT0000 complete [ 4231.234170] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 4238.220225] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4245.699187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4251.055335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4256.025696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4260.775874] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4266.860784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4266.919103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4267.152115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4267.182733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4267.254808] Lustre: lustre-MDT0000: new disk, initializing [ 4267.341333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4267.364817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4270.788362] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4280.102760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4280.188267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4280.313077] Lustre: 169045:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4280.407554] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4280.410465] Lustre: Skipped 1 previous similar message [ 4280.519503] Lustre: lustre-MDT0001: new disk, initializing [ 4280.694081] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4280.744229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4280.760470] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4280.841307] Lustre: 169841:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 4280.847334] Lustre: 169841:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 4280.853182] Lustre: 169841:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 4280.867735] Lustre: 169841:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 4280.878621] Lustre: 169841:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 4280.885385] Lustre: 169841:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 4283.960246] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4291.664862] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4295.501947] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4303.439321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4303.527250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4303.903794] Lustre: lustre-OST0000: new disk, initializing [ 4303.913208] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4304.041072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4309.193861] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4309.587129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4309.599958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4309.715453] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4315.714427] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4319.026682] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4320.472279] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 15:55:53 (1761249353) [ 4530.145440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4530.147451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4530.159795] Lustre: Skipped 1 previous similar message [ 4530.169777] Lustre: Skipped 2 previous similar messages [ 4534.778160] LustreError: 172566:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4534.780706] LustreError: 172566:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4534.896230] Lustre: server umount lustre-MDT0000 complete [ 4535.264461] LustreError: 169056:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4535.295378] LustreError: 169056:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 4538.339355] LustreError: 169834:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761249571 with bad export cookie 17392033120603745209 [ 4538.342610] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4538.349810] LustreError: 169834:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4540.385251] LustreError: 169052:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4540.389226] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4540.402639] Lustre: Skipped 1 previous similar message [ 4540.406848] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4543.393924] LustreError: 169052:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4543.402192] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4544.534918] LustreError: 172770:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4544.541706] LustreError: 172770:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4544.721118] Lustre: server umount lustre-MDT0001 complete [ 4562.400175] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4562.465776] LustreError: 172973:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4562.482097] LustreError: 172973:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4562.759410] Lustre: server umount lustre-OST0000 complete [ 4573.131695] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 4582.610521] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4591.815937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4598.314389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4603.210543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4609.752060] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4617.228167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4617.292724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4617.517924] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4617.550442] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4617.606620] Lustre: lustre-MDT0000: new disk, initializing [ 4617.725761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4617.758930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4620.795720] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4636.145103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4636.230951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4636.302805] Lustre: 175316:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4636.345867] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4636.348811] Lustre: Skipped 1 previous similar message [ 4636.419210] Lustre: lustre-MDT0001: new disk, initializing [ 4636.562520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4636.600045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4636.609408] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4640.642684] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4647.160814] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4650.053982] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4655.433603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4655.497136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4655.723757] Lustre: lustre-OST0000: new disk, initializing [ 4655.736359] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4655.823888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4660.750395] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4665.920771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4665.937596] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4666.040066] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4667.255214] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4670.626690] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4670.785349] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4673.775788] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4673.967670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4676.577186] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4676.593891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4676.594777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4676.624085] Lustre: Skipped 1 previous similar message [ 4676.690314] Lustre: Skipped 1 previous similar message [ 4681.665352] LustreError: 178268:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4681.667893] LustreError: 178268:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4681.700113] LustreError: 177061:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4681.724032] LustreError: 177061:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 4681.830788] Lustre: server umount lustre-OST0000 complete [ 4686.821225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4686.837073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4686.841047] Lustre: Skipped 1 previous similar message [ 4690.717118] LustreError: 178469:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4690.719689] LustreError: 178469:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4690.779193] Lustre: server umount lustre-MDT0000 complete [ 4691.941093] LustreError: 175322:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4691.951594] LustreError: 175322:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 4693.555189] LustreError: 175308:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761249726 with bad export cookie 17392033120603904081 [ 4693.561485] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4693.908090] Lustre: server umount lustre-MDT0001 complete [ 4700.887729] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:02:13 (1761249733) [ 4713.601827] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 4721.105360] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4728.440417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4734.391468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4740.576456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4746.080395] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4751.718743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4751.753626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4751.907477] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4751.932498] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4752.013105] Lustre: lustre-MDT0000: new disk, initializing [ 4752.096446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4752.113964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4755.215229] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4764.753444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4764.820311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4765.001077] Lustre: 181660:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4765.317461] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4765.425796] Lustre: 182462:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 4765.430347] Lustre: 182462:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 4765.470482] Lustre: 182462:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 4765.494062] Lustre: 182462:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4765.498102] Lustre: 182462:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 4765.526449] Lustre: 182462:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4765.542372] Lustre: 182462:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 4765.554816] Lustre: 182462:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4765.558453] Lustre: 182462:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 4765.562148] Lustre: 182462:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4765.565939] Lustre: 182462:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 4765.596340] Lustre: 182462:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4768.859674] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4776.546486] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4780.144192] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4786.552435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4786.629916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4786.810321] Lustre: lustre-OST0000: new disk, initializing [ 4786.812537] Lustre: Skipped 1 previous similar message [ 4786.819214] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4786.822020] Lustre: Skipped 2 previous similar messages [ 4792.235185] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4796.543697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4796.554892] Lustre: Skipped 1 previous similar message [ 4796.566032] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4796.682556] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4799.268405] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4807.139046] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4807.141724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4807.142183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4807.142187] LustreError: Skipped 1 previous similar message [ 4807.152336] Lustre: Skipped 2 previous similar messages [ 4810.935278] LustreError: 184209:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4810.947389] LustreError: 184209:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4811.078056] Lustre: server umount lustre-OST0000 complete [ 4816.353173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4816.358370] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4820.407782] Lustre: server umount lustre-MDT0000 complete [ 4822.498071] LustreError: 181668:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4822.510619] LustreError: 181668:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 4823.388384] LustreError: 181652:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761249856 with bad export cookie 17392033120603905005 [ 4823.394106] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4823.403475] LustreError: 181652:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4823.679594] Lustre: server umount lustre-MDT0001 complete [ 4829.984548] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:04:22 (1761249862) [ 4843.195529] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 4851.511630] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4860.386797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4866.411681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4872.444485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4878.447912] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4886.967582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4887.053117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4887.295258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4887.344805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4887.512138] Lustre: lustre-MDT0000: new disk, initializing [ 4887.621292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4887.624645] Lustre: Skipped 2 previous similar messages [ 4887.652510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4891.781174] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4901.676123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4901.743924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4901.853438] Lustre: 187607:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4902.163615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4902.266404] LustreError: 188411:0:(llog_cat.c:511:llog_cat_current_log()) lustre-MDT0001-osd: next log does not exist, catalog [0x240000400:0x1:0x0] rc=-5 [ 4902.272074] LustreError: 188411:0:(update_trans.c:965:top_trans_stop()) lustre-MDT0001-osd: write updates failed: rc = -5 [ 4902.281033] LustreError: 188411:0:(mdt_handler.c:8424:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 4902.332263] Lustre: 188411:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 515 < left 524, rollback = 2 [ 4902.350609] Lustre: 188411:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 4902.381879] Lustre: 188411:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/0, destroy: 0/0/0 [ 4902.394737] Lustre: 188411:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4902.402777] Lustre: 188411:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 4902.406632] Lustre: 188411:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4902.409655] Lustre: 188411:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 4902.413352] Lustre: 188411:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4902.417093] Lustre: 188411:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/1, delete: 0/0/0 [ 4902.420579] Lustre: 188411:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4902.424222] Lustre: 188411:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 4902.428105] Lustre: 188411:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 4905.478205] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4911.482897] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4914.242284] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4919.374957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4919.477272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4923.673259] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4927.591636] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4929.083299] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4937.702751] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4937.704940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4937.720884] LustreError: Skipped 1 previous similar message [ 4937.722216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4937.726889] Lustre: Skipped 2 previous similar messages [ 4937.735942] Lustre: Skipped 5 previous similar messages [ 4940.540101] LustreError: 190156:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4940.547249] LustreError: 190156:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4940.673473] Lustre: server umount lustre-OST0000 complete [ 4958.179344] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4958.373370] Lustre: server umount lustre-MDT0000 complete [ 4961.309833] LustreError: 187599:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761249994 with bad export cookie 17392033120603906132 [ 4961.312119] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4961.339840] LustreError: 187599:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4961.669030] Lustre: server umount lustre-MDT0001 complete [ 4968.492360] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 4970.379197] Key type lgssc unregistered [ 4970.606664] LNet: 191093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4970.611374] LNetError: 191093:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4970.620834] LNet: Removed LNI 192.168.201.103@tcp [ 4971.179273] Key type .llcrypt unregistered [ 4971.180837] Key type ._llcrypt unregistered [ 4991.965267] Key type ._llcrypt registered [ 4991.967048] Key type .llcrypt registered [ 4992.066654] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4992.576771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4992.609811] alg: No test for adler32 (adler32-zlib) [ 4993.636641] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 4993.818605] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 4995.480191] Key type lgssc registered [ 4996.464686] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5003.285192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5003.299902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5004.749305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5008.144614] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5014.998917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5015.010476] LustreError: 192491:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5015.035640] LustreError: 192491:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 5015.296299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5019.322714] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5023.554038] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5026.372428] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5031.734486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5032.007054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5036.230588] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5040.245136] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5050.407900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5070.817428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5070.823549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5070.830321] LustreError: Skipped 1 previous similar message [ 5070.844781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5070.847198] Lustre: Skipped 1 previous similar message [ 5072.903165] LustreError: 194766:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5073.090584] Lustre: server umount lustre-OST0000 complete [ 5076.961629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5076.967951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5076.978091] Lustre: Skipped 1 previous similar message [ 5076.982294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5081.057405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5081.064915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5081.084794] Lustre: Skipped 1 previous similar message [ 5082.430639] LustreError: 194968:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5082.433481] LustreError: 194968:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5082.544289] Lustre: server umount lustre-MDT0000 complete [ 5085.705745] LustreError: 192473:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761250119 with bad export cookie 8789588830196185864 [ 5085.713321] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5085.728184] LustreError: 192473:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5085.875624] LustreError: 195169:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5085.881829] LustreError: 195169:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5086.077457] Lustre: server umount lustre-MDT0001 complete [ 5094.613597] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5097.044311] Key type lgssc unregistered [ 5097.302760] LNet: 195701:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5097.314247] LNetError: 195701:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5097.336813] LNet: Removed LNI 192.168.201.103@tcp [ 5097.942366] Key type .llcrypt unregistered [ 5097.944458] Key type ._llcrypt unregistered [ 5113.825874] Key type ._llcrypt registered [ 5113.827642] Key type .llcrypt registered [ 5113.896397] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5114.590682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5114.649918] alg: No test for adler32 (adler32-zlib) [ 5115.650793] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 5115.816308] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5117.464141] Key type lgssc registered [ 5118.443882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5124.741332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5124.760884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5126.094826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5128.974858] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5135.206453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5135.507219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5138.594363] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5142.881352] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5145.417312] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5150.576162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5150.873569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5153.286721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5155.344647] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5159.580768] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5165.537314] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5165.541581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5165.553954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5165.559385] Lustre: Skipped 1 previous similar message [ 5165.566227] Lustre: Skipped 1 previous similar message [ 5166.375976] Lustre: lustre-OST0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 5170.428113] LustreError: 199326:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5170.575329] Lustre: server umount lustre-OST0000 complete [ 5175.146122] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff994ac765f800.ost_server_uuid 50 [ 5176.389797] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5191.137705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5191.143033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5191.145131] Lustre: Skipped 1 previous similar message [ 5191.154501] Lustre: Skipped 1 previous similar message [ 5194.863670] LustreError: 199696:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5194.866805] LustreError: 199696:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5194.954314] Lustre: server umount lustre-MDT0000 complete [ 5196.257388] LustreError: 197077:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5196.279485] LustreError: 197077:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 5198.271050] LustreError: 197057:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761250231 with bad export cookie 6573772627662161479 [ 5198.275844] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5198.278521] LustreError: 197057:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5198.391124] LustreError: 199898:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5198.394717] LustreError: 199898:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5198.560694] Lustre: server umount lustre-MDT0001 complete [ 5205.623504] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 16:10:38 (1761250238) [ 5211.082638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5211.469798] LustreError: 200570:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5211.542227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5214.825516] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5216.737942] LustreError: 200570:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5220.836633] LustreError: 200570:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5221.068390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5221.335952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5224.638494] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5228.806287] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5231.862487] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5236.584560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5236.823029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5241.443131] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5246.062780] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5251.964493] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5252.015346] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5252.136841] Lustre: lustre-OST0001: new disk, initializing [ 5252.139830] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5256.376681] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5256.960603] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5260.885797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5260.895599] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5260.979297] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5263.139397] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5266.412256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5266.436259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5266.444179] Lustre: Skipped 1 previous similar message [ 5271.366371] LustreError: 203776:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5271.370641] LustreError: 203776:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5271.404992] Lustre: server umount lustre-OST0000 complete [ 5271.527405] LustreError: 203599:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5271.534641] LustreError: 203599:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 5276.134301] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5276.316291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5279.874694] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5280.057248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5281.764160] LustreError: 202170:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5281.774882] LustreError: 202170:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 5 previous similar messages [ 5297.120894] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5297.126773] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5297.133637] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5297.133644] Lustre: Skipped 1 previous similar message [ 5297.187353] Lustre: Skipped 2 previous similar messages [ 5299.592580] LustreError: 204499:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5299.601711] LustreError: 204499:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5299.968637] Lustre: server umount lustre-OST0001 complete [ 5307.369567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5307.376661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5307.386354] Lustre: Skipped 1 previous similar message [ 5307.391796] Lustre: Skipped 1 previous similar message [ 5310.719913] LustreError: 204701:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5310.724229] LustreError: 204701:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5310.831221] Lustre: server umount lustre-MDT0000 complete [ 5312.484056] LustreError: 200567:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5312.505426] LustreError: 200567:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 9 previous similar messages [ 5314.260917] LustreError: 200552:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761250347 with bad export cookie 6573772627662162550 [ 5314.272864] LustreError: 200552:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5314.278422] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5314.656525] Lustre: server umount lustre-MDT0001 complete [ 5320.177312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5323.588156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5327.213247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5331.022783] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5336.295864] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 16:12:49 (1761250369) [ 5341.977813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5342.295299] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5342.327055] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5342.480300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5342.482706] Lustre: Skipped 1 previous similar message [ 5346.350987] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5354.358474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5354.433737] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5354.467664] Lustre: 206395:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5358.035745] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5362.537957] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5365.946810] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5371.636605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5371.763909] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5376.710760] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5378.687781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5381.807170] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5386.742870] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5386.857804] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5387.046497] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5387.053607] Lustre: Skipped 2 previous similar messages [ 5388.598152] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5391.005583] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5395.791387] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5399.018857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5399.021345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5399.023127] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5399.023234] Lustre: Skipped 1 previous similar message [ 5399.032934] Lustre: Skipped 2 previous similar messages [ 5412.320249] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5412.367248] LustreError: 209597:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5412.378657] LustreError: 209597:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5412.482272] Lustre: server umount lustre-OST0000 complete [ 5414.375646] LustreError: 208030:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5414.404256] LustreError: 208030:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 5429.730491] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5429.732097] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5429.742354] LustreError: Skipped 1 previous similar message [ 5433.806389] Lustre: server umount lustre-OST0001 complete [ 5439.971222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5439.973787] Lustre: Skipped 9 previous similar messages [ 5444.009693] Lustre: server umount lustre-MDT0000 complete [ 5447.719063] LustreError: 206389:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761250481 with bad export cookie 6573772627662163957 [ 5447.721255] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5447.744252] LustreError: 206389:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5447.864354] LustreError: 210225:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5447.869630] LustreError: 210225:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 5448.073349] Lustre: server umount lustre-MDT0001 complete [ 5453.565329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5457.264414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5461.602605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5465.886254] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5472.644974] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 16:15:05 (1761250505) [ 5486.423911] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 5494.502885] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5504.021635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5510.366688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5516.094771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5521.243909] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5527.552160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5527.628192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5527.808743] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5527.856727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5527.935753] Lustre: lustre-MDT0000: new disk, initializing [ 5528.055647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5528.072927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5531.678955] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5542.524946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5542.593951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5542.648449] Lustre: 214053:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5542.707495] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5542.710595] Lustre: Skipped 1 previous similar message [ 5542.763821] Lustre: lustre-MDT0001: new disk, initializing [ 5542.893845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5542.915346] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5546.697898] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5552.737546] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5556.138971] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5562.019359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5562.074909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5562.237978] Lustre: lustre-OST0000: new disk, initializing [ 5562.244163] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5566.719299] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5570.087040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5570.094690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5570.178461] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5572.533996] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5575.454869] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5575.613859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5578.337453] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5578.491032] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5580.259352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5580.261723] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5580.278845] Lustre: Skipped 3 previous similar messages [ 5580.281387] LustreError: Skipped 1 previous similar message [ 5580.282970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5580.293425] Lustre: Skipped 1 previous similar message [ 5586.053416] LustreError: 216989:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5586.060343] LustreError: 216989:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5586.214965] Lustre: server umount lustre-OST0000 complete [ 5595.629661] LustreError: 216991:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5595.659312] LustreError: 216991:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 10 previous similar messages [ 5595.737241] Lustre: server umount lustre-MDT0000 complete [ 5599.636856] LustreError: 214044:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761250633 with bad export cookie 6573772627662165742 [ 5599.639408] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5599.651597] LustreError: 214044:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5600.010682] Lustre: server umount lustre-MDT0001 complete [ 5607.217666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5611.086944] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5619.018925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5622.516408] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5627.367326] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5630.607118] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5635.868520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5640.516161] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5644.727228] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5648.515864] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5653.059123] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 5656.522774] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5656.695768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5659.619888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5659.624254] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5659.630815] Lustre: Skipped 2 previous similar messages [ 5664.604490] Lustre: server umount lustre-OST0000 complete [ 5670.071097] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5673.820745] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5702.442553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5702.797259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5702.807317] Lustre: Skipped 5 previous similar messages [ 5708.279655] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5708.297684] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5709.295279] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5713.387118] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5713.396721] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5715.346551] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5721.256749] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 5726.350135] LustreError: 222286:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5726.358786] LustreError: 222286:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 5726.510765] Lustre: server umount lustre-OST0000 complete [ 5732.321103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5732.330780] LustreError: Skipped 1 previous similar message [ 5732.342974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5732.346534] Lustre: Skipped 8 previous similar messages [ 5740.147922] Lustre: server umount lustre-MDT0000 complete [ 5743.776066] LustreError: 217850:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761250777 with bad export cookie 6573772627662166673 [ 5743.779305] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5743.808486] LustreError: 217850:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5754.553482] Lustre: server umount lustre-MDT0001 complete [ 5762.431558] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 16:19:55 (1761250795) [ 5769.733826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5770.370904] LustreError: 223363:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5770.392415] LustreError: 223363:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 4 previous similar messages [ 5774.129867] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5782.145275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5786.516690] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5791.080903] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5793.914881] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5799.536978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5805.044129] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5810.222269] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5813.851906] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5816.191430] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5819.338901] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5819.551513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5825.541600] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5825.620668] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5825.738705] Lustre: lustre-OST0001: new disk, initializing [ 5825.755688] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5830.498593] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5831.243407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5831.253711] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5831.318205] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5836.314581] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5839.482771] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5839.651659] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5842.417212] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5842.571919] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5847.018230] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5847.018690] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5847.029915] LustreError: Skipped 1 previous similar message [ 5847.038171] Lustre: Skipped 4 previous similar messages [ 5858.273350] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5858.411639] Lustre: server umount lustre-OST0001 complete [ 5863.170145] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5863.344153] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5866.678512] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5866.883544] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5894.293351] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5899.766281] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5899.781681] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5899.782942] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5900.029742] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5905.336457] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5907.114859] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5923.296238] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5923.363627] Lustre: server umount lustre-OST0001 complete [ 5940.193981] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5940.197590] LustreError: Skipped 2 previous similar messages [ 5944.615706] Lustre: server umount lustre-OST0000 complete [ 5953.681557] Lustre: server umount lustre-MDT0000 complete [ 5956.063110] LustreError: 223343:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761250989 with bad export cookie 6573772627662167639 [ 5956.066901] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5956.075063] LustreError: 223343:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5970.400172] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5970.664295] Lustre: server umount lustre-MDT0001 complete [ 5974.995615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5978.007257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5981.091880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5984.038975] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5989.285821] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 16:23:42 (1761251022) [ 5994.348884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5994.552689] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5994.578904] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5994.712573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5994.715848] Lustre: Skipped 5 previous similar messages [ 5997.736364] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6003.858478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6003.960086] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6004.008365] Lustre: 231365:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6007.430823] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6012.181303] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6015.484813] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6021.236169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6021.419830] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6026.089328] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6030.535450] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6038.563553] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6038.621511] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6043.587409] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6049.111846] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6050.662112] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6053.902833] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6054.116851] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6057.196049] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6057.366917] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6060.091240] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff994aca616800.ost_server_uuid 50 [ 6061.274463] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff994aca616800.ost_server_uuid in FULL state after 0 sec [ 6062.248851] Lustre: Permanently deactivating lustre-OST0001 [ 6062.259980] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6064.724227] Lustre: Permanently reactivating lustre-OST0001 [ 6069.445518] LustreError: 235354:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6069.449893] LustreError: 235354:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 6069.495087] Lustre: server umount lustre-OST0001 complete [ 6072.290839] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6072.310140] LustreError: 232992:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6072.324255] LustreError: 232992:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 33 previous similar messages [ 6073.830815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6073.855278] Lustre: Skipped 23 previous similar messages [ 6078.954801] Lustre: server umount lustre-OST0000 complete [ 6088.682166] Lustre: server umount lustre-MDT0000 complete [ 6091.232599] LustreError: 235202:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761251124 with bad export cookie 6573772627662168927 [ 6091.234793] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6091.239951] LustreError: 235202:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6091.527100] Lustre: server umount lustre-MDT0001 complete [ 6095.498399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6098.346589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6101.062256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6103.941989] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6109.137622] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 16:25:42 (1761251142) [ 6110.179674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6115.319740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6115.523533] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6115.561377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6115.585917] Lustre: Skipped 5 previous similar messages [ 6119.157840] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6126.629647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6126.706535] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6126.748108] Lustre: 237515:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6130.184371] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6134.241973] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6136.890200] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6141.837600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6141.914413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6142.027427] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6142.051766] Lustre: Permanently deactivating lustre-OST0000 [ 6142.065662] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6143.469427] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6147.580600] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6152.613179] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6157.168608] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6157.243397] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6162.635587] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6166.692107] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6167.148728] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6168.571681] Lustre: Permanently reactivating lustre-OST0000 [ 6168.584607] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6168.588136] Lustre: Skipped 2 previous similar messages [ 6174.788683] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6174.797765] Lustre: Skipped 12 previous similar messages [ 6174.803267] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 32 seconds. I think it's dead, and I am evicting it. exp 000000004ca1dfc8, cur 1761251208 deadline 1761251206 last 1761251176 [ 6174.804311] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6174.825980] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6174.839603] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6174.849358] Lustre: Skipped 1 previous similar message [ 6174.887476] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6185.532192] Lustre: server umount lustre-OST0001 complete [ 6194.753335] Lustre: server umount lustre-OST0000 complete [ 6203.847824] Lustre: server umount lustre-MDT0000 complete [ 6206.951422] LustreError: 237505:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761251240 with bad export cookie 6573772627662170572 [ 6206.952948] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6206.975689] LustreError: 237505:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6207.397433] Lustre: server umount lustre-MDT0001 complete [ 6214.612201] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 6216.720657] Key type lgssc unregistered [ 6216.921901] LNet: 241912:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6216.929958] LNetError: 241912:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6216.943656] LNet: Removed LNI 192.168.201.103@tcp [ 6217.533611] Key type .llcrypt unregistered [ 6217.538213] Key type ._llcrypt unregistered [ 6233.617323] Key type ._llcrypt registered [ 6233.619238] Key type .llcrypt registered [ 6233.683578] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 6234.169815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6234.180722] alg: No test for adler32 (adler32-zlib) [ 6235.103338] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 6235.238620] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 6236.872190] Key type lgssc registered [ 6237.587128] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6239.797633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6244.435032] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6244.454281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6245.797461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6248.503320] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6254.723325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6254.781158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6254.886380] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6254.921685] Lustre: 243328:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6254.929852] Lustre: Permanently deactivating lustre-MDT0001 [ 6254.940970] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6255.220598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6258.436355] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6262.317257] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6262.857326] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6266.172119] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6271.544022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6271.830956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6277.002476] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6280.104907] LustreError: 244963:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6280.138836] LustreError: 244963:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 6280.183622] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6282.743214] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6285.288040] LustreError: 244962:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6288.920200] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6289.169408] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6291.271128] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6291.281546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6291.299866] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6293.907697] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6298.241299] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6300.582457] Lustre: Permanently reactivating lustre-MDT0001 [ 6300.596877] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6300.602044] Lustre: Skipped 2 previous similar messages [ 6305.583962] Lustre: lustre-MDT0001: Client 5b42e1fc-26e9-49f9-9446-649e5c5388a2 (at 192.168.201.3@tcp) reconnecting [ 6308.551462] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6309.468842] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6309.488392] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 6309.494683] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6309.506171] LustreError: 246580:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 6309.508531] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6309.511331] LustreError: 246580:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 6309.511500] LustreError: 246580:0:(obd_config.c:1997:class_config_llog_handler()) MGC192.168.201.103@tcp: cfg command failed: rc = -108 [ 6309.511518] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 6309.511518] [ 6309.517213] LustreError: 243332:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6309.804944] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6313.778264] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6313.941499] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6316.804914] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6316.962203] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6317.926393] Lustre: 244475:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [ 6317.932040] Lustre: 244475:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6317.939168] Lustre: 244475:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [ 6317.942938] Lustre: 244475:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 7/100/0, punch: 0/0/0, quota 1/3/0 [ 6317.950748] Lustre: 244475:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6317.954926] Lustre: 244475:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6318.845730] Lustre: Permanently deactivating lustre-MDT0001 [ 6318.889028] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6318.892122] Lustre: Skipped 2 previous similar messages [ 6319.423250] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6326.283679] Lustre: 243336:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 506 < left 979, rollback = 2 [ 6326.288756] Lustre: 243336:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 6326.293349] Lustre: 243336:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 2/8/6, destroy: 0/0/0 [ 6326.296779] Lustre: 243336:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6326.300250] Lustre: 243336:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 6/979/0 [ 6326.307800] Lustre: 243336:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6326.315299] Lustre: 243336:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 6326.322753] Lustre: 243336:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6326.327421] Lustre: 243336:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 8/130/4, delete: 0/0/0 [ 6326.331725] Lustre: 243336:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6326.335613] Lustre: 243336:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 4/4/0, ref_del: 0/0/0 [ 6326.340900] Lustre: 243336:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 6329.831853] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6329.835949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6329.853492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6333.416518] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6333.417575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6333.425417] Lustre: Skipped 2 previous similar messages [ 6333.431404] Lustre: Skipped 2 previous similar messages [ 6335.085394] LustreError: 247413:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6335.169923] Lustre: server umount lustre-MDT0000 complete [ 6338.057518] LustreError: 243321:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761251371 with bad export cookie 14678958642042258019 [ 6338.067890] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6338.083867] LustreError: 243321:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6338.209909] LustreError: 247615:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 6338.221970] LustreError: 247615:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6338.377021] Lustre: server umount lustre-MDT0001 complete [ 6347.454802] LustreError: 247815:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6347.466971] LustreError: 247815:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6347.524175] Lustre: server umount lustre-OST0000 complete [ 6356.744458] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 16:29:49 (1761251389) [ 6365.416985] LustreError: 248822:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6365.420670] LustreError: 248822:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6365.576263] Lustre: server umount lustre-OST0001 complete [ 6370.493462] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 6376.527797] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 6383.208337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6388.126293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6393.221735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6399.190146] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6404.753057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6404.809782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6405.001867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6405.022758] Lustre: Skipped 2 previous similar messages [ 6405.064708] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6405.165506] Lustre: lustre-MDT0000: new disk, initializing [ 6405.278888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6405.307651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6408.888504] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6417.224984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6417.270533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6417.331660] Lustre: 251007:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6417.386928] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6417.392804] Lustre: Skipped 1 previous similar message [ 6417.454884] Lustre: lustre-MDT0001: new disk, initializing [ 6417.535990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6417.564022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6417.576196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6417.640883] Lustre: 251805:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 6417.652908] Lustre: 251805:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 3 previous similar messages [ 6417.663809] Lustre: 251805:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6417.680500] Lustre: 251805:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 6417.684217] Lustre: 251805:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 6417.697082] Lustre: 251805:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 6417.713386] Lustre: 251805:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 6417.723895] Lustre: 251805:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 6417.734599] Lustre: 251805:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6417.742214] Lustre: 251805:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 6417.751605] Lustre: 251805:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6417.755009] Lustre: 251805:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 6420.782307] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6426.923136] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6430.147525] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6436.901751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6436.981617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6437.229254] Lustre: lustre-OST0000: new disk, initializing [ 6437.239787] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6437.332374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6443.071071] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6443.087033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6443.094760] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6443.181116] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6449.870883] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6451.637084] LustreError: 253364:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6459.383188] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6459.427849] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6459.545714] Lustre: lustre-OST0001: new disk, initializing [ 6459.557513] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6461.683629] LustreError: 253364:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6461.731654] LustreError: 253364:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6465.071976] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6467.143896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6467.155736] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6467.258252] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6471.054108] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6471.769056] LustreError: 253364:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6472.195118] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6475.569312] Lustre: lustre-OST0001: Not available for connect from 192.168.201.3@tcp (stopping) [ 6477.797477] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6477.808495] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6477.824808] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6479.737177] LustreError: 254504:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6479.742878] LustreError: 254504:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6479.804300] Lustre: server umount lustre-OST0001 complete [ 6480.673741] LustreError: 252740:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6482.921249] LustreError: 252740:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6482.942557] LustreError: 252740:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 6488.039988] LustreError: 252739:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6488.047221] LustreError: 252739:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 6498.284802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6498.287468] LustreError: 252741:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6498.288918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6498.288925] Lustre: Skipped 1 previous similar message [ 6498.298715] Lustre: Skipped 2 previous similar messages [ 6498.319944] LustreError: 252741:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 6502.153126] LustreError: 254729:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6502.162149] LustreError: 254729:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6502.330474] Lustre: server umount lustre-OST0000 complete [ 6508.514908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6508.536702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6508.539339] Lustre: Skipped 1 previous similar message [ 6511.663670] Lustre: server umount lustre-MDT0000 complete [ 6514.686182] LustreError: 252756:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761251548 with bad export cookie 14678958642042260833 [ 6514.688840] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6514.698623] LustreError: 252756:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6520.899282] LustreError: 255131:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6520.905349] LustreError: 255131:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6521.049254] Lustre: server umount lustre-MDT0001 complete [ 6528.699141] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 6530.873857] Key type lgssc unregistered [ 6531.159905] LNet: 255663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6531.166979] LNetError: 255663:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6531.178629] LNet: Removed LNI 192.168.201.103@tcp [ 6531.839454] Key type .llcrypt unregistered [ 6531.841228] Key type ._llcrypt unregistered [ 6558.361234] Key type ._llcrypt registered [ 6558.364752] Key type .llcrypt registered [ 6558.459686] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 6572.330148] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 6573.363726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6573.392191] alg: No test for adler32 (adler32-zlib) [ 6574.384513] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 6574.587626] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 6576.256192] Key type lgssc registered [ 6577.258774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6584.630445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6589.520855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6594.194084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6599.419880] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6604.077994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6604.131350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6604.141661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6605.351695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6605.386194] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6605.472901] Lustre: lustre-MDT0000: new disk, initializing [ 6605.560521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6605.570644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6608.451612] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6616.865701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6616.932439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6617.019871] Lustre: 259616:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6617.077974] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6617.083482] Lustre: Skipped 1 previous similar message [ 6617.148907] Lustre: lustre-MDT0001: new disk, initializing [ 6617.262444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6617.308427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6617.319498] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6617.384875] Lustre: 260411:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 6617.401095] Lustre: 260411:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 6617.408117] Lustre: 260411:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 6617.422270] Lustre: 260411:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 6617.431522] Lustre: 260411:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 6617.439690] Lustre: 260411:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 6620.676773] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6626.370461] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6629.005619] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6633.903320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6633.957493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6634.160860] Lustre: lustre-OST0000: new disk, initializing [ 6634.182443] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6634.255565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6638.609453] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6643.765203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6643.771577] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6643.879458] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6644.102493] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6647.094462] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6647.254729] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6650.267530] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6650.434860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6654.439190] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6654.439913] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6654.443460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6654.451722] Lustre: Skipped 1 previous similar message [ 6654.466548] LustreError: Skipped 1 previous similar message [ 6659.559288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6659.561837] Lustre: Skipped 2 previous similar messages [ 6664.683223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6666.208153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6666.262083] LustreError: 262553:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6666.445583] Lustre: server umount lustre-OST0000 complete [ 6669.793664] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6669.800696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6669.809735] Lustre: Skipped 1 previous similar message [ 6674.918930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6674.929502] Lustre: Skipped 1 previous similar message [ 6675.051903] LustreError: 262757:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6675.056018] LustreError: 262757:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6675.124486] Lustre: server umount lustre-MDT0000 complete [ 6678.784593] LustreError: 261361:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761251712 with bad export cookie 8214292624690217474 [ 6678.789355] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6678.862179] LustreError: 261361:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6678.966774] LustreError: 262958:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6678.978487] LustreError: 262958:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6679.420070] Lustre: server umount lustre-MDT0001 complete [ 6687.446653] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 16:35:20 (1761251720) [ 6693.267186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6694.044682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6697.696357] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6699.492781] LustreError: 263629:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6699.510531] LustreError: 263629:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 6704.610950] LustreError: 263629:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6707.127475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6707.419286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6710.337934] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6714.726101] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6717.448580] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6722.532499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6722.777332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6726.995589] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6731.089648] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6785.508712] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6785.508961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6785.523859] Lustre: Skipped 1 previous similar message [ 6785.528116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6785.535847] Lustre: Skipped 1 previous similar message [ 6791.164255] LustreError: 265979:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6791.171660] LustreError: 265979:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6791.421338] Lustre: server umount lustre-OST0000 complete [ 6794.897096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6804.996039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6805.522040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6810.432074] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6814.213679] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6815.783490] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6815.796792] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6815.798530] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6815.832607] Lustre: Skipped 1 previous similar message [ 6851.555839] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6851.567688] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6851.577144] Lustre: Skipped 1 previous similar message [ 6851.580465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6851.585045] Lustre: Skipped 3 previous similar messages [ 6857.398364] LustreError: 267643:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6857.409932] LustreError: 267643:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6857.580623] Lustre: server umount lustre-OST0000 complete [ 6861.282264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6861.296292] LustreError: Skipped 1 previous similar message [ 6861.309417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6861.329489] Lustre: Skipped 1 previous similar message [ 6866.712737] LustreError: 267845:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6866.724803] LustreError: 267845:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6866.802334] Lustre: server umount lustre-MDT0000 complete [ 6866.914140] LustreError: 265247:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6866.945069] LustreError: 265247:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 6869.849763] LustreError: 263610:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761251903 with bad export cookie 8214292624690218398 [ 6869.871955] LustreError: 263610:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6869.872955] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6870.241823] Lustre: server umount lustre-MDT0001 complete [ 6878.460973] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 6880.961528] Key type lgssc unregistered [ 6881.263836] LNet: 268577:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6881.272842] LNetError: 268577:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6881.289870] LNet: Removed LNI 192.168.201.103@tcp [ 6881.997905] Key type .llcrypt unregistered [ 6882.001424] Key type ._llcrypt unregistered [ 6901.172281] Key type ._llcrypt registered [ 6901.174021] Key type .llcrypt registered [ 6901.269070] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 6901.840497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6901.914570] alg: No test for adler32 (adler32-zlib) [ 6902.875508] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 6903.019156] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 6904.659315] Key type lgssc registered [ 6905.445619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6910.903529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6910.922451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6912.276654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6914.979983] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6921.214313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6921.584110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6924.768423] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6929.118679] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6931.866819] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6937.237958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6937.519114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6941.897238] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6946.514778] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6956.975789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6977.511036] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6977.512084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6977.518720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6977.524494] Lustre: Skipped 1 previous similar message [ 6982.629220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6982.642599] Lustre: Skipped 1 previous similar message [ 6983.300167] LustreError: 272996:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6983.464365] Lustre: server umount lustre-OST0000 complete [ 6987.749616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6987.753089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6987.765279] Lustre: Skipped 1 previous similar message [ 6987.770203] Lustre: Skipped 1 previous similar message [ 6992.492441] LustreError: 273197:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6992.521538] LustreError: 273197:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6992.622060] Lustre: server umount lustre-MDT0000 complete [ 6992.867080] LustreError: 269956:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6992.878295] LustreError: 269956:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 6995.542765] LustreError: 269936:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761252028 with bad export cookie 7258558523583848382 [ 6995.549275] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6995.556102] LustreError: 269936:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6995.739497] LustreError: 273398:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6995.746500] LustreError: 273398:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6996.126234] Lustre: server umount lustre-MDT0001 complete [ 7004.650141] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7006.726608] Key type lgssc unregistered [ 7007.006266] LNet: 273931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7007.012768] LNetError: 273931:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7007.035275] LNet: Removed LNI 192.168.201.103@tcp [ 7007.656132] Key type .llcrypt unregistered [ 7007.658576] Key type ._llcrypt unregistered [ 7020.303684] Key type ._llcrypt registered [ 7020.305818] Key type .llcrypt registered [ 7020.392803] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7021.061600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7021.084878] alg: No test for adler32 (adler32-zlib) [ 7022.161109] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 7022.351261] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 7024.040648] Key type lgssc registered [ 7024.942080] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7031.390464] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7031.470946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7033.011214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7036.558751] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7043.568751] LustreError: 275113:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7043.617316] LustreError: 275113:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 7045.229273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7045.710894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7049.976391] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7055.787393] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7058.802834] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7065.547275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7065.951518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7071.311931] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7076.295274] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7078.268660] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7093.729026] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7093.737105] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7093.755505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7093.769193] Lustre: Skipped 1 previous similar message [ 7094.733153] LustreError: 277507:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7095.051398] Lustre: server umount lustre-OST0000 complete [ 7101.923523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7101.930465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7101.964912] Lustre: Skipped 1 previous similar message [ 7101.972135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7103.969876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7103.971095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7105.247226] LustreError: 277709:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7105.254973] LustreError: 277709:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7105.333499] Lustre: server umount lustre-MDT0000 complete [ 7108.699530] LustreError: 276735:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761252142 with bad export cookie 8049590075650754317 [ 7108.704668] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7108.882899] LustreError: 277911:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7108.896846] LustreError: 277911:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7109.121374] Lustre: server umount lustre-MDT0001 complete [ 7119.596634] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7122.603460] Key type lgssc unregistered [ 7122.869050] LNet: 278443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7122.876491] LNetError: 278443:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7122.888250] LNet: Removed LNI 192.168.201.103@tcp [ 7123.666489] Key type .llcrypt unregistered [ 7123.668264] Key type ._llcrypt unregistered [ 7148.054235] Key type ._llcrypt registered [ 7148.055771] Key type .llcrypt registered [ 7148.167928] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7149.375271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7149.387351] alg: No test for adler32 (adler32-zlib) [ 7150.512194] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 7150.810157] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 7152.488172] Key type lgssc registered [ 7153.668542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7160.240778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7160.268368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7161.692780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7164.796137] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7172.066449] LustreError: 279819:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7172.084095] LustreError: 279819:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 7172.503654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7172.917584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7176.870842] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7182.332556] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7186.271766] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7193.753881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7194.144396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7200.014640] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7205.991328] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7207.487068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7238.121357] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7238.122279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7238.126870] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7238.132720] Lustre: Skipped 1 previous similar message [ 7240.293102] LustreError: 282844:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7240.567612] Lustre: server umount lustre-OST0000 complete [ 7244.769306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7244.774914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7244.806108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7244.808546] Lustre: Skipped 1 previous similar message [ 7248.353717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7248.356967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7250.440566] LustreError: 283045:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7250.458092] LustreError: 283045:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7250.524956] Lustre: server umount lustre-MDT0000 complete [ 7253.475524] LustreError: 281666:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7253.507746] LustreError: 281666:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 7254.300883] LustreError: 279799:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761252287 with bad export cookie 2755093347517702939 [ 7254.329664] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7254.456805] LustreError: 283249:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7254.483542] LustreError: 283249:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7254.814585] Lustre: server umount lustre-MDT0001 complete [ 7264.318158] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7267.115311] Key type lgssc unregistered [ 7267.432374] LNet: 283781:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7267.439367] LNetError: 283781:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7267.476767] LNet: Removed LNI 192.168.201.103@tcp [ 7268.184496] Key type .llcrypt unregistered [ 7268.186281] Key type ._llcrypt unregistered [ 7285.008704] Key type ._llcrypt registered [ 7285.015333] Key type .llcrypt registered [ 7285.098353] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7286.471955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7286.517505] alg: No test for adler32 (adler32-zlib) [ 7287.569907] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 7287.759486] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 7289.408227] Key type lgssc registered [ 7290.479190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7297.910461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7297.926517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7299.565134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7303.447206] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7308.793096] LustreError: 285009:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7308.800672] LustreError: 285009:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 7312.112832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7312.596850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7316.485557] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7321.601806] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7325.342976] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7331.460299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7331.841511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7337.043208] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7342.047432] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7344.175527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7359.461042] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7359.463211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7359.476442] Lustre: Skipped 1 previous similar message [ 7359.511883] Lustre: Skipped 1 previous similar message [ 7360.496088] LustreError: 287413:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7360.680452] Lustre: server umount lustre-OST0000 complete [ 7364.580139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7364.620405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7369.704157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7369.711754] Lustre: Skipped 2 previous similar messages [ 7370.789770] LustreError: 287615:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7370.809988] LustreError: 287615:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7370.882288] Lustre: server umount lustre-MDT0000 complete [ 7374.616861] LustreError: 284965:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761252407 with bad export cookie 16630781747221076158 [ 7374.622838] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7374.638149] LustreError: 284965:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7374.785137] LustreError: 287817:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7374.790934] LustreError: 287817:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7375.087930] Lustre: server umount lustre-MDT0001 complete [ 7385.283264] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7388.169987] Key type lgssc unregistered [ 7388.478610] LNet: 288350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7388.491836] LNetError: 288350:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7388.517828] LNet: Removed LNI 192.168.201.103@tcp [ 7389.514572] Key type .llcrypt unregistered [ 7389.516245] Key type ._llcrypt unregistered [ 7398.636283] Key type ._llcrypt registered [ 7398.643147] Key type .llcrypt registered [ 7398.761234] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7414.311141] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 7431.322279] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7431.958548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7432.068223] alg: No test for adler32 (adler32-zlib) [ 7433.079730] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 7433.305413] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 7434.960184] Key type lgssc registered [ 7436.118480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7444.764763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7452.442989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7459.546846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7467.594947] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7474.294445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7474.341230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7474.354413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7475.619812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7475.691817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7475.764796] Lustre: lustre-MDT0000: new disk, initializing [ 7475.882311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7475.900113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7479.610409] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7490.417879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7490.478487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7490.643352] Lustre: 291861:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7490.820886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7490.844756] Lustre: Skipped 1 previous similar message [ 7490.998447] Lustre: lustre-MDT0001: new disk, initializing [ 7491.125040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7491.168105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7491.183769] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7491.323122] Lustre: 292660:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 7491.346831] Lustre: 292660:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 7491.355688] Lustre: 292660:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 7491.374393] Lustre: 292660:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 7491.383068] Lustre: 292660:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 7491.392434] Lustre: 292660:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 7495.302763] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7502.263746] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7505.534630] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7512.388362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7512.484630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7512.830967] Lustre: lustre-OST0000: new disk, initializing [ 7512.840846] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7512.956692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7518.664244] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7521.385208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7521.398585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7521.546522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7526.733767] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7531.038466] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7531.223709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7536.055523] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7536.402476] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7542.243868] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7542.246089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7542.273884] LustreError: Skipped 1 previous similar message [ 7542.274824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7542.286735] Lustre: Skipped 1 previous similar message [ 7544.980218] LustreError: 294803:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7545.213301] Lustre: server umount lustre-OST0000 complete [ 7552.481316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7552.488146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7552.490915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7552.490922] Lustre: Skipped 1 previous similar message [ 7552.516810] Lustre: Skipped 1 previous similar message [ 7556.302447] LustreError: 295005:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7556.318912] LustreError: 295005:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7556.492614] Lustre: server umount lustre-MDT0000 complete [ 7560.644278] LustreError: 291852:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761252594 with bad export cookie 9088173005740864136 [ 7560.657802] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7560.668764] LustreError: 291852:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7560.806726] LustreError: 295206:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7560.829088] LustreError: 295206:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7561.232261] Lustre: server umount lustre-MDT0001 complete [ 7569.366181] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 16:50:02 (1761252602) [ 7575.913952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7576.363330] LustreError: 295878:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7576.383620] LustreError: 295878:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 7576.444831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7580.832409] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7581.670151] LustreError: 295877:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7585.786169] LustreError: 295877:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7590.482915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7590.829347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7594.966917] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7600.260907] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7603.910441] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7611.697822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7612.152601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7617.493992] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7622.691894] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7631.841867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7631.851032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7631.860756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7631.873949] Lustre: Skipped 2 previous similar messages [ 7636.963059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7636.970792] Lustre: Skipped 1 previous similar message [ 7637.127286] LustreError: 298128:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7637.131635] LustreError: 298128:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7637.272586] Lustre: server umount lustre-OST0000 complete [ 7642.083137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7642.083172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7642.140407] Lustre: Skipped 1 previous similar message [ 7642.156063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7652.332666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7652.337595] Lustre: Skipped 4 previous similar messages [ 7655.393265] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7655.474896] LustreError: 298330:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7655.477179] LustreError: 298330:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7655.567510] Lustre: server umount lustre-MDT0000 complete [ 7657.442076] LustreError: 296180:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7657.456784] LustreError: 296180:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 7658.949758] LustreError: 295858:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761252692 with bad export cookie 9088173005740865053 [ 7658.953572] LustreError: 295858:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7658.956953] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7659.286430] Lustre: server umount lustre-MDT0001 complete [ 7668.484733] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7671.308609] Key type lgssc unregistered [ 7671.645659] LNet: 299068:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7671.651838] LNetError: 299068:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7671.661566] LNet: Removed LNI 192.168.201.103@tcp [ 7672.542343] Key type .llcrypt unregistered [ 7672.543894] Key type ._llcrypt unregistered [ 7697.605254] Key type ._llcrypt registered [ 7697.606948] Key type .llcrypt registered [ 7697.738989] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 7711.349307] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7712.418633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7712.517373] alg: No test for adler32 (adler32-zlib) [ 7713.580076] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 7713.803513] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 7715.481611] Key type lgssc registered [ 7716.693727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7725.612152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7731.747895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7737.173740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7743.411910] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7748.070084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7753.275768] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7759.980178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7760.057127] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7760.085739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7761.336357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7761.394654] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7761.454458] Lustre: lustre-MDT0000: new disk, initializing [ 7761.557193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7761.581809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7765.618899] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7776.930824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7777.008965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7777.092295] Lustre: 302877:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7777.162164] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7777.164961] Lustre: Skipped 1 previous similar message [ 7777.247316] Lustre: lustre-MDT0001: new disk, initializing [ 7777.396336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7777.441275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7777.459336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7777.550782] Lustre: 303677:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 7777.571647] Lustre: 303677:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 7777.582431] Lustre: 303677:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 7777.618587] Lustre: 303677:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 7777.633926] Lustre: 303677:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 7777.646608] Lustre: 303677:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 7781.600409] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7788.699205] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7791.606891] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7797.646805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7797.700899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7797.870339] Lustre: lustre-OST2710: new disk, initializing [ 7797.876674] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 7797.970076] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7802.851356] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7805.507318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7805.525714] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7805.620079] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 7809.631471] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 7815.893801] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7815.997738] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7816.244980] Lustre: lustre-OST03e8: new disk, initializing [ 7816.256979] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 7816.346749] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7821.348253] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7825.475787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7825.480617] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7825.524365] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 7826.407923] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 7832.023494] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 7832.189962] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7837.192519] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 7837.423496] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7841.211520] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 7841.470361] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7846.250145] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 7846.489807] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7850.979168] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7850.991511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7850.997471] Lustre: Skipped 1 previous similar message [ 7851.006884] Lustre: Skipped 3 previous similar messages [ 7852.359343] LustreError: 307224:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7852.451555] Lustre: server umount lustre-MDT0000 complete [ 7855.639304] LustreError: 302871:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761252889 with bad export cookie 11403745106313470559 [ 7855.644859] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7855.667156] LustreError: 302871:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7855.881836] LustreError: 307425:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7855.884358] LustreError: 307425:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7856.136389] Lustre: server umount lustre-MDT0001 complete [ 7870.225151] LustreError: 307626:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7870.246109] LustreError: 307626:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7870.367619] Lustre: server umount lustre-OST2710 complete [ 7872.288162] Lustre: 300655:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761252889/real 1761252889] req@0000000002698c8f x1846807360501248/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761252905 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7872.308292] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7872.328729] Lustre: Skipped 2 previous similar messages [ 7874.162550] LustreError: 307828:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7874.203329] LustreError: 307828:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7874.408409] Lustre: server umount lustre-OST03e8 complete [ 7892.418583] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 7902.405984] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7911.786304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7917.922525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7923.628483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7929.424619] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7936.451718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7936.518634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7936.728483] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7936.775601] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7936.863121] Lustre: lustre-MDT0000: new disk, initializing [ 7936.973250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7936.999087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7941.008588] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7951.167250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7951.260346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7951.428577] Lustre: 310625:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7951.492826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7951.495913] Lustre: Skipped 1 previous similar message [ 7951.550788] Lustre: lustre-MDT0001: new disk, initializing [ 7951.683390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7951.745443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7951.771870] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7955.212880] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7962.498544] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7965.426366] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7971.944570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7972.015844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7972.288184] Lustre: lustre-OST0000: new disk, initializing [ 7972.301131] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7972.397345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7977.993520] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7978.544301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7978.549951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7978.678803] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7984.888307] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7989.833829] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7990.077029] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7994.287228] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7994.474985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7998.945953] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7998.957245] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7998.989858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8004.580769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8004.584246] Lustre: Skipped 2 previous similar messages [ 8009.701335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8009.722254] Lustre: Skipped 1 previous similar message [ 8011.232158] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8011.298357] LustreError: 313567:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8011.301228] LustreError: 313567:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8011.466267] Lustre: server umount lustre-OST0000 complete [ 8018.403273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8018.406836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8018.415976] Lustre: Skipped 1 previous similar message [ 8018.420279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8018.424778] Lustre: Skipped 1 previous similar message [ 8022.488118] LustreError: 313771:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8022.509123] LustreError: 313771:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8022.717964] Lustre: server umount lustre-MDT0000 complete [ 8026.468706] LustreError: 311408:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761253059 with bad export cookie 11403745106313472288 [ 8026.471765] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8026.484747] LustreError: 311408:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8026.962189] Lustre: server umount lustre-MDT0001 complete [ 8036.157627] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 16:57:48 (1761253068) [ 8037.827638] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8040.137270] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 16:57:52 (1761253072) [ 8043.529393] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8052.670343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8057.187294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8062.447086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8066.761203] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8067.892941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8075.976670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8076.342228] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8076.388436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8076.558750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8080.365310] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8089.004326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8089.289482] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8089.352322] Lustre: 316107:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8093.653067] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8099.448620] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8103.073952] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8109.310443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8109.432551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8109.608027] Lustre: Denying initial registration attempt from nid 192.168.201.103@tcp, specified as failover [ 8109.620987] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8109.625449] LustreError: 317727:0:(tgt_mount.c:2482:server_fill_super()) Unable to start targets: -99 [ 8109.631775] LustreError: 317727:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 8109.642994] LustreError: 317727:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 8109.741906] Lustre: server umount lustre-OST0000 complete [ 8109.747560] LustreError: 317727:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8116.193515] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8116.198318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8116.217585] Lustre: Skipped 2 previous similar messages [ 8116.235657] Lustre: Skipped 3 previous similar messages [ 8121.689505] LustreError: 318095:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 8121.702188] LustreError: 318095:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8121.842180] Lustre: server umount lustre-MDT0000 complete [ 8125.539194] LustreError: 316099:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761253158 with bad export cookie 11403745106313473212 [ 8125.551320] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8125.900675] Lustre: server umount lustre-MDT0001 complete [ 8135.177988] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 8137.860285] Key type lgssc unregistered [ 8138.127914] LNet: 318829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8138.138416] LNetError: 318829:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8138.171333] LNet: Removed LNI 192.168.201.103@tcp [ 8139.007348] Key type .llcrypt unregistered [ 8139.009149] Key type ._llcrypt unregistered [ 8149.608588] Key type ._llcrypt registered [ 8149.611463] Key type .llcrypt registered [ 8149.715526] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8150.359886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8150.369887] alg: No test for adler32 (adler32-zlib) [ 8151.476642] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 8151.701446] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 8153.384198] Key type lgssc registered [ 8154.365508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8161.401529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8165.356906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8169.049779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8172.722193] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8173.789996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8188.336927] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8196.668275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8196.695963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8198.080303] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8198.103237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8198.234447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8201.729884] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8209.688588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8209.797121] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8209.817462] Lustre: 321602:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8210.066135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8213.695089] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8218.442512] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8221.827635] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8227.648029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8227.773427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8227.996402] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8228.181705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8233.138966] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8238.396215] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8242.169281] LustreError: 323900:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8242.306922] Lustre: server umount lustre-OST0000 complete [ 8245.732249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8245.740511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8245.749419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8250.348443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8250.352454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8255.461693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8255.483684] Lustre: Skipped 2 previous similar messages [ 8256.501838] LustreError: 324102:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8256.519162] LustreError: 324102:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8256.613065] Lustre: server umount lustre-MDT0000 complete [ 8260.206442] LustreError: 321596:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761253293 with bad export cookie 413238540423041966 [ 8260.208444] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8260.218355] LustreError: 321596:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8269.807889] LustreError: 324304:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8269.818575] LustreError: 324304:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8270.048315] Lustre: server umount lustre-MDT0001 complete [ 8281.024180] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 8283.417462] Key type lgssc unregistered [ 8283.743553] LNet: 324838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8283.761934] LNetError: 324838:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8284.776940] LNet: Removed LNI 192.168.201.103@tcp [ 8285.634132] Key type .llcrypt unregistered [ 8285.635901] Key type ._llcrypt unregistered [ 8307.895524] Key type ._llcrypt registered [ 8307.901622] Key type .llcrypt registered [ 8308.033517] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8309.550721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8309.571121] alg: No test for adler32 (adler32-zlib) [ 8310.649688] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 8310.925727] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 8312.648206] Key type lgssc registered [ 8314.288987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8322.880508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8322.895924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8324.273514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8328.393221] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8333.794678] LustreError: 326234:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8333.810469] LustreError: 326234:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 8337.783460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8338.186547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8341.868430] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8346.727036] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8350.148826] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8356.370306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8356.709506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8361.982493] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8368.259703] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8379.368635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8379.381348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8379.384630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8379.404820] Lustre: Skipped 2 previous similar messages [ 8381.224517] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 8382.642264] LustreError: 328490:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8382.745328] Lustre: server umount lustre-MDT0000 complete [ 8384.485795] LustreError: 326234:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8384.501478] LustreError: 326234:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 8385.958647] LustreError: 326215:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761253419 with bad export cookie 1648042766671989150 [ 8385.962321] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8385.984183] LustreError: 326215:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8386.327669] LustreError: 326229:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8386.359536] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.3@tcp (stopping) [ 8389.602041] LustreError: 327295:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8389.604692] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8389.627177] Lustre: Skipped 2 previous similar messages [ 8389.631288] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8395.749106] LustreError: 327295:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8395.749782] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8395.762524] LustreError: 327295:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 8395.783702] Lustre: Skipped 2 previous similar messages [ 8400.352154] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8400.426427] LustreError: 328692:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8400.429095] LustreError: 328692:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8400.666643] Lustre: server umount lustre-MDT0001 complete [ 8407.503201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8414.777435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8431.849516] LustreError: 327849:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8431.870664] LustreError: 327849:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 8431.912322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8431.960405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8435.535383] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8442.777240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8443.206078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8446.872673] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8451.565410] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8451.802902] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8453.186393] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8455.772191] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8456.855512] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8461.818284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8461.825114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8461.838174] Lustre: Skipped 1 previous similar message [ 8461.853330] Lustre: Skipped 2 previous similar messages [ 8466.592752] LustreError: 331003:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8466.611393] LustreError: 331003:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8466.808341] Lustre: server umount lustre-OST0000 complete [ 8472.036521] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8472.048387] Lustre: Skipped 1 previous similar message [ 8476.288824] LustreError: 331204:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8476.294964] LustreError: 331204:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8476.393356] Lustre: server umount lustre-MDT0000 complete [ 8477.154713] LustreError: 329508:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8477.173520] LustreError: 329508:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 8479.724897] LustreError: 326215:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761253513 with bad export cookie 1648042766671996815 [ 8479.726700] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8479.733246] LustreError: 326215:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8480.114722] Lustre: server umount lustre-MDT0001 complete [ 8489.254163] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 8491.740660] Key type lgssc unregistered [ 8492.018688] LNet: 331936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8492.025474] LNetError: 331936:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8492.045409] LNet: Removed LNI 192.168.201.103@tcp [ 8492.834595] Key type .llcrypt unregistered [ 8492.841610] Key type ._llcrypt unregistered [ 8517.081429] Key type ._llcrypt registered [ 8517.082933] Key type .llcrypt registered [ 8517.239265] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8518.800251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8518.812246] alg: No test for adler32 (adler32-zlib) [ 8519.858226] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 8520.117304] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 8521.816167] Key type lgssc registered [ 8522.887019] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8529.228752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8529.240105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8530.727603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8534.199272] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8541.159479] LustreError: 333333:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8541.169538] LustreError: 333333:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 8542.076303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8542.385439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8545.853532] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8551.136798] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8554.917598] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8561.125405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8561.128636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8561.153941] Lustre: Skipped 1 previous similar message [ 8564.838457] LustreError: 334726:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8564.942823] Lustre: server umount lustre-MDT0000 complete [ 8566.241728] LustreError: 333333:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8566.258691] LustreError: 333333:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 8567.265173] LustreError: 334046:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8567.286265] LustreError: 334046:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 8567.861588] LustreError: 334728:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761253601 with bad export cookie 14363705230246004341 [ 8567.871081] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8567.875789] LustreError: 334728:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8567.989351] LustreError: 334929:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8567.998961] LustreError: 334929:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8568.184931] Lustre: server umount lustre-MDT0001 complete [ 8579.102643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8579.331341] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8579.356520] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8579.514795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8582.486590] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8589.294476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8589.434074] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8589.480292] Lustre: 335698:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8589.711220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8593.184651] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8597.674841] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8600.546370] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8607.566594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8607.736884] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8607.749785] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8607.760853] LustreError: 337358:0:(tgt_mount.c:2482:server_fill_super()) Unable to start targets: -2 [ 8607.765916] LustreError: 337358:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 8607.771872] LustreError: 337358:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 8607.883307] Lustre: server umount lustre-OST0000 complete [ 8607.891472] LustreError: 337358:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8616.659441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8616.866748] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8617.097945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8623.687839] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8623.867023] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 8629.057647] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8635.895335] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8635.964690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8636.029332] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8636.097821] Lustre: lustre-OST0001: new disk, initializing [ 8636.102169] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8636.205983] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8641.059909] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8642.637507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8642.652320] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8642.735121] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8645.011704] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8648.162774] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8648.166321] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8648.166501] LustreError: Skipped 1 previous similar message [ 8648.169884] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8648.175299] Lustre: Skipped 1 previous similar message [ 8648.192669] Lustre: Skipped 2 previous similar messages [ 8653.244129] LustreError: 339483:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 8653.261622] LustreError: 339483:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8653.297028] LustreError: 338189:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8653.302162] LustreError: 338189:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 8653.353820] Lustre: server umount lustre-OST0001 complete [ 8658.404707] LustreError: 338189:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8658.405331] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8658.406372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8658.421876] LustreError: 338189:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [ 8658.431644] Lustre: Skipped 1 previous similar message [ 8662.198452] LustreError: 339684:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8662.203275] LustreError: 339684:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8662.352715] Lustre: server umount lustre-OST0000 complete [ 8666.593370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8666.597088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8666.604923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8666.607475] Lustre: Skipped 1 previous similar message [ 8671.537149] LustreError: 339886:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8671.545957] LustreError: 339886:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8671.608191] Lustre: server umount lustre-MDT0000 complete [ 8673.763827] LustreError: 337850:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8674.265667] LustreError: 335690:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761253707 with bad export cookie 14363705230246004901 [ 8674.271430] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8674.286227] LustreError: 335690:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8674.595207] Lustre: server umount lustre-MDT0001 complete [ 8684.061854] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 8686.950466] Key type lgssc unregistered [ 8687.289447] LNet: 340619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8687.293638] LNetError: 340619:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8688.358694] LNet: Removed LNI 192.168.201.103@tcp [ 8689.428205] Key type .llcrypt unregistered [ 8689.436828] Key type ._llcrypt unregistered [ 8717.214514] Key type ._llcrypt registered [ 8717.218071] Key type .llcrypt registered [ 8717.289042] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 8731.151143] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8732.215308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8732.241119] alg: No test for adler32 (adler32-zlib) [ 8733.308459] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 8733.567372] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 8735.273295] Key type lgssc registered [ 8736.311795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8743.383149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8748.157923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8752.659804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8757.636053] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8762.741853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8762.795721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8762.810297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8764.114682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8764.169285] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8764.238199] Lustre: lustre-MDT0000: new disk, initializing [ 8764.318609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8764.341200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8767.624817] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8776.890181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8776.954864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8777.018417] Lustre: 344560:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8777.074230] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8777.078243] Lustre: Skipped 1 previous similar message [ 8777.131972] Lustre: lustre-MDT0001: new disk, initializing [ 8777.214152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8777.261919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8777.276453] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8777.334805] Lustre: 345360:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 8777.340850] Lustre: 345360:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 8777.345129] Lustre: 345360:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 8777.351031] Lustre: 345360:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 8777.359959] Lustre: 345360:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 8777.370060] Lustre: 345360:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 8780.777576] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8787.631785] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8790.930641] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8795.988848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8796.058318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8796.359521] Lustre: lustre-OST0000: new disk, initializing [ 8796.364385] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8796.448756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8800.813929] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8803.397769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8803.404119] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8803.472964] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8806.945937] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8810.272839] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8810.481852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8813.768512] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8813.990704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8819.173282] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8819.174035] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8819.176498] LustreError: Skipped 1 previous similar message [ 8819.187476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8821.568761] LustreError: 347503:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8821.741285] Lustre: server umount lustre-OST0000 complete [ 8824.289735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8824.295846] Lustre: Skipped 1 previous similar message [ 8824.298873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8824.301921] Lustre: Skipped 2 previous similar messages [ 8829.409120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8829.412059] Lustre: Skipped 1 previous similar message [ 8830.355609] LustreError: 347705:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8830.359782] LustreError: 347705:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8830.431586] Lustre: server umount lustre-MDT0000 complete [ 8833.325503] LustreError: 344555:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761253866 with bad export cookie 12347861122671554349 [ 8833.327800] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8833.344792] LustreError: 344555:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8833.451281] LustreError: 347906:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8833.454646] LustreError: 347906:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8833.631684] Lustre: server umount lustre-MDT0001 complete [ 8840.147210] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 17:11:12 (1761253872) [ 8843.998202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8848.716379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8866.364001] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 8872.356737] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8879.509205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8884.671955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8889.689533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8894.890261] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8900.818841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8900.885705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8901.092724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8901.137887] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8901.214722] Lustre: lustre-MDT0000: new disk, initializing [ 8901.297512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8901.312054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8904.776141] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8914.569675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8914.609887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8914.681443] Lustre: 351681:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8914.727690] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8914.730781] Lustre: Skipped 1 previous similar message [ 8914.775830] Lustre: lustre-MDT0001: new disk, initializing [ 8914.861210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8914.902857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8914.917430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8914.979627] Lustre: 352477:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 8914.987129] Lustre: 352477:0:(osd_internal.h:1361:osd_trans_exec_op()) Skipped 1 previous similar message [ 8914.990738] Lustre: 352477:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 8914.994025] Lustre: 352477:0:(osd_handler.c:2088:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8914.997363] Lustre: 352477:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 8915.000903] Lustre: 352477:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8915.003786] Lustre: 352477:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 8915.025116] Lustre: 352477:0:(osd_handler.c:2105:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8915.035366] Lustre: 352477:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 8915.047256] Lustre: 352477:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8915.062952] Lustre: 352477:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 8915.072617] Lustre: 352477:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message [ 8918.149616] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8924.387296] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8927.651489] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8933.412716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8933.485841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8933.932085] Lustre: lustre-OST0000: new disk, initializing [ 8933.935153] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8934.018574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8939.063535] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8940.589243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8940.597680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8940.703651] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8945.614459] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8949.348679] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8949.504553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8952.787727] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8952.984589] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8956.390970] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8956.393545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8956.398856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8956.407328] Lustre: Skipped 2 previous similar messages [ 8956.423869] LustreError: Skipped 1 previous similar message [ 8960.630152] LustreError: 354622:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8960.633835] LustreError: 354622:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8960.834686] Lustre: server umount lustre-OST0000 complete [ 8966.114784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8966.118731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8966.126213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8966.129040] Lustre: Skipped 1 previous similar message [ 8975.842681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8975.853958] Lustre: Skipped 4 previous similar messages [ 8978.400180] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8978.503293] LustreError: 354824:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8978.511451] LustreError: 354824:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8980.650332] Lustre: server umount lustre-MDT0000 complete [ 8984.306112] LustreError: 351672:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761254017 with bad export cookie 12347861122671555273 [ 8984.309522] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8984.401632] LustreError: 351672:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8984.826062] Lustre: server umount lustre-MDT0001 complete [ 8993.363718] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 17:13:46 (1761254026) [ 8999.975234] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 17:13:52 (1761254032) [ 9009.431300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9009.803934] LustreError: 356043:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9009.816525] LustreError: 356043:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 9009.877569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9013.722805] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9015.270032] LustreError: 356043:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9020.390513] LustreError: 356042:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9021.810614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9025.545449] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9030.312804] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9033.724667] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9040.351769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9046.973879] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9052.846207] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9064.735223] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9066.162822] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9067.522621] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9068.877416] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9070.184085] Lustre: Failing over lustre-MDT0000 [ 9070.338599] LustreError: 358507:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9070.353137] LustreError: 358507:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 9070.412866] Lustre: server umount lustre-MDT0000 complete [ 9071.075427] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9071.081549] LustreError: 356038:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9071.098331] Lustre: Skipped 2 previous similar messages [ 9071.116755] LustreError: 356038:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 9076.194045] LustreError: 356042:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9076.212952] LustreError: 356042:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 9084.692392] LustreError: 356039:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9084.707208] LustreError: 356039:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 4 previous similar messages [ 9087.460496] Lustre: 342778:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761254104/real 1761254104] req@00000000764033b9 x1846808429046400/t0(0) o400->MGC192.168.201.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761254120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9087.476262] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9087.714495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9096.688566] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xab5c6a44f0b1ba7f [ 9096.704778] Lustre: MGC192.168.201.103@tcp: Connection restored to 0@lo (at 0@lo) [ 9096.882084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9096.887081] Lustre: Skipped 2 previous similar messages [ 9096.936351] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9098.060873] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9100.439309] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9102.322718] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9102.387547] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9102.435915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9108.246281] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9110.017267] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9113.201414] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9114.843433] Lustre: Failing over lustre-MDT0000 [ 9114.984587] LustreError: 359589:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9115.001717] LustreError: 359589:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9115.125144] Lustre: server umount lustre-MDT0000 complete [ 9117.667966] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9117.671432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9117.672428] LustreError: 356043:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9117.672435] LustreError: 356043:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 14 previous similar messages [ 9117.681203] Lustre: Skipped 3 previous similar messages [ 9132.545316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9132.682722] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9132.963455] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9133.850279] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9136.420696] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9138.173405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9138.179101] Lustre: Skipped 2 previous similar messages [ 9138.192227] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9138.274504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9143.409001] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9145.035310] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9153.505298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9153.508894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9153.533392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9153.572637] Lustre: Skipped 2 previous similar messages [ 9156.134739] LustreError: 360622:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9156.138822] LustreError: 360622:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9156.273186] Lustre: server umount lustre-MDT0000 complete [ 9158.633361] LustreError: 356037:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9158.647361] LustreError: 356037:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 13 previous similar messages [ 9159.594690] LustreError: 356763:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761254192 with bad export cookie 12347861122671558017 [ 9159.598052] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9159.610022] LustreError: 356763:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9159.960115] Lustre: server umount lustre-MDT0001 complete [ 9173.798766] Lustre: server umount lustre-OST0000 complete [ 9187.449758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9188.049933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9188.059269] Lustre: Skipped 1 previous similar message [ 9191.643890] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9199.693906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9204.183233] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9209.505809] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9212.894506] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9219.809588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9224.711398] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9229.498875] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9233.418432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9238.498083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9238.506842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9238.510854] Lustre: Skipped 1 previous similar message [ 9238.516198] Lustre: Skipped 2 previous similar messages [ 9240.382161] LustreError: 364104:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9240.391110] LustreError: 364104:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 9240.542722] Lustre: server umount lustre-OST0000 complete [ 9246.185857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9249.940380] Lustre: server umount lustre-MDT0000 complete [ 9253.513857] LustreError: 361836:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761254286 with bad export cookie 12347861122671558752 [ 9253.523917] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9253.524457] LustreError: 361836:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9253.944567] Lustre: server umount lustre-MDT0001 complete [ 9263.110508] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 9265.439663] Key type lgssc unregistered [ 9265.804416] LNet: 365038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9265.818678] LNetError: 365038:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9265.835965] LNet: Removed LNI 192.168.201.103@tcp [ 9266.904823] Key type .llcrypt unregistered [ 9266.906683] Key type ._llcrypt unregistered [ 9292.836378] Key type ._llcrypt registered [ 9292.839333] Key type .llcrypt registered [ 9292.980694] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 9293.935811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9293.968377] alg: No test for adler32 (adler32-zlib) [ 9295.305946] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 9295.660417] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 9297.392104] Key type lgssc registered [ 9298.646913] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9307.624779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9307.664044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9309.300073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9313.381271] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9318.892223] LustreError: 366534:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9318.915395] LustreError: 366534:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 9321.700310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9322.010850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9325.390988] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9330.422835] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9333.737980] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9339.858899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9340.117879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9345.133591] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9350.175587] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9354.488026] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9364.964992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9364.969643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9364.979201] Lustre: Skipped 2 previous similar messages [ 9364.985038] Lustre: Skipped 2 previous similar messages [ 9366.366318] LustreError: 368809:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9366.640264] Lustre: server umount lustre-MDT0000 complete [ 9370.083067] LustreError: 368701:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9370.098018] LustreError: 368701:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 9370.784039] LustreError: 366515:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761254404 with bad export cookie 8346673922993505367 [ 9370.792768] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9370.839122] LustreError: 366515:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9370.957143] LustreError: 369010:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9370.966552] LustreError: 369010:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9371.135993] Lustre: server umount lustre-MDT0001 complete [ 9383.709796] LustreError: 369211:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9383.716179] LustreError: 369211:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9384.015319] Lustre: server umount lustre-OST0000 complete [ 9397.727555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9398.335274] LustreError: 369944:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9398.457818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9402.349893] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9403.874910] LustreError: 369944:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9407.974068] LustreError: 369944:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9410.676132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9411.055455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9414.203437] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9417.989636] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9421.171246] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9428.004561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9428.320717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9433.008825] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9436.840368] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9442.669627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9462.245412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9462.257853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9462.300464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9463.271591] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9463.282638] Lustre: Skipped 1 previous similar message [ 9465.573128] LustreError: 372313:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9465.580036] LustreError: 372313:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9465.754855] Lustre: server umount lustre-MDT0000 complete [ 9468.389931] LustreError: 371454:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9468.405399] LustreError: 371454:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [ 9469.792561] LustreError: 369924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761254503 with bad export cookie 8346673922993506480 [ 9469.807847] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9469.820041] LustreError: 369924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9469.969587] LustreError: 372515:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9469.977763] LustreError: 372515:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9470.206028] Lustre: server umount lustre-MDT0001 complete [ 9483.604457] LustreError: 372716:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9483.607588] LustreError: 372716:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9483.820920] Lustre: server umount lustre-OST0000 complete [ 9496.040256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9496.426853] LustreError: 373544:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9496.508870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9499.530689] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9506.695986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9510.489777] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9515.004728] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9518.127637] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9524.370054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9529.403813] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9533.795790] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9542.130437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9552.352936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9552.356131] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9552.373812] LustreError: Skipped 1 previous similar message [ 9552.413183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9552.416022] Lustre: Skipped 4 previous similar messages [ 9554.058941] LustreError: 375815:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9554.068743] LustreError: 375815:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9554.269629] Lustre: server umount lustre-OST0000 complete [ 9557.990187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9558.003491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9558.022336] Lustre: Skipped 1 previous similar message [ 9558.045935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9562.596151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9562.607422] Lustre: Skipped 1 previous similar message [ 9563.727329] Lustre: server umount lustre-MDT0000 complete [ 9566.354392] LustreError: 373525:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761254599 with bad export cookie 8346673922993507495 [ 9566.357796] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9566.365736] LustreError: 373525:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9566.660024] Lustre: server umount lustre-MDT0001 complete [ 9574.359095] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 9576.710967] Key type lgssc unregistered [ 9576.935205] LNet: 376750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9576.940961] LNetError: 376750:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9576.953977] LNet: Removed LNI 192.168.201.103@tcp [ 9577.599971] Key type .llcrypt unregistered [ 9577.602100] Key type ._llcrypt unregistered [ 9597.921809] Key type ._llcrypt registered [ 9597.926320] Key type .llcrypt registered [ 9598.005164] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 9598.681765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9598.798947] alg: No test for adler32 (adler32-zlib) [ 9599.901138] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 9600.108262] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 9601.776263] Key type lgssc registered [ 9602.779853] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9609.674190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9609.691152] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9609.697501] LustreError: 378150:0:(osd_handler.c:8747:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9609.714610] LustreError: 378150:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9609.719535] LustreError: 378150:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9609.723572] LustreError: 378150:0:(tgt_mount.c:2450:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9609.728661] LustreError: 378150:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9616.002395] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9616.010389] LustreError: 378548:0:(osd_handler.c:8747:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9616.034538] LustreError: 378548:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9616.041902] LustreError: 378548:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9616.051043] LustreError: 378548:0:(tgt_mount.c:2450:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9616.058433] LustreError: 378548:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9632.885576] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 9635.364194] Key type lgssc unregistered [ 9635.594357] LNet: 379536:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9635.613583] LNetError: 379536:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9635.629711] LNet: Removed LNI 192.168.201.103@tcp [ 9636.405590] Key type .llcrypt unregistered [ 9636.409095] Key type ._llcrypt unregistered [ 9650.952082] Key type ._llcrypt registered [ 9650.958905] Key type .llcrypt registered [ 9651.103151] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 9665.766414] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 9666.394893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9666.418479] alg: No test for adler32 (adler32-zlib) [ 9667.602456] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 9667.875916] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 9669.616157] Key type lgssc registered [ 9670.588719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9676.998686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9681.617889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9686.118265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9690.592394] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9696.192264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9696.261306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9696.274035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9697.541751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9697.572643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9697.622563] Lustre: lustre-MDT0000: new disk, initializing [ 9697.708696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9697.726817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9701.039609] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9711.139231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9711.232375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9711.386427] Lustre: 382665:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9711.469778] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9711.472976] Lustre: Skipped 1 previous similar message [ 9711.565618] Lustre: lustre-MDT0001: new disk, initializing [ 9711.698282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9711.753671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9711.758966] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9711.834441] Lustre: 383461:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [ 9711.840088] Lustre: 383461:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [ 9711.846938] Lustre: 383461:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [ 9711.855782] Lustre: 383461:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [ 9711.862439] Lustre: 383461:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [ 9711.872309] Lustre: 383461:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [ 9715.826964] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9722.234169] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9725.096187] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9730.551923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9730.617911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9730.796519] Lustre: lustre-OST0000: new disk, initializing [ 9730.806077] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9730.906775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9735.907579] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9740.321892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9740.328840] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9740.416586] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9742.067970] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9746.103123] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9746.315942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9750.474451] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9750.648132] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9755.621842] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9755.631339] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9755.644456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9756.132853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9756.135637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9756.179050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9756.180926] Lustre: Skipped 1 previous similar message [ 9758.603201] LustreError: 385607:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9758.872346] Lustre: server umount lustre-OST0000 complete [ 9762.785334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9762.793879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9762.812751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9766.378259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9766.386149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9771.491217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9771.511786] Lustre: Skipped 2 previous similar messages [ 9777.120683] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9777.305740] LustreError: 385808:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9777.400295] LustreError: 385808:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9777.672896] Lustre: server umount lustre-MDT0000 complete [ 9782.126824] LustreError: 382657:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761254815 with bad export cookie 586837523596183633 [ 9782.128210] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9782.134324] LustreError: 382657:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9782.333989] LustreError: 386013:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9782.341355] LustreError: 386013:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9782.556248] Lustre: server umount lustre-MDT0001 complete [ 9791.711988] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9793.223900] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 17:27:06 (1761254826) [ 9799.113757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9799.494915] LustreError: 386735:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9799.506884] LustreError: 386735:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 9799.571895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9803.503898] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9804.771814] LustreError: 386735:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9808.865438] LustreError: 386735:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9810.470482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9810.734496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9813.517555] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9818.519051] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9821.544771] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9826.887325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9827.131622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9831.316255] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9835.170878] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9840.367746] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9840.435909] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9840.680277] Lustre: lustre-OST0001: new disk, initializing [ 9840.684568] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9841.619217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9841.625552] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9841.728174] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9845.478914] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9850.914764] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9853.927151] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9853.929485] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9853.934302] Lustre: Skipped 1 previous similar message [ 9853.937422] Lustre: Skipped 2 previous similar messages [ 9859.715176] LustreError: 389935:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 9859.723864] LustreError: 389935:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9859.818543] Lustre: server umount lustre-OST0001 complete [ 9862.424856] LustreError: 388331:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9869.288809] LustreError: 388333:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9869.308961] LustreError: 388333:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 9877.475035] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9877.484213] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9877.504038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9877.506737] Lustre: Skipped 3 previous similar messages [ 9879.530117] LustreError: 389752:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9879.536400] LustreError: 389752:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [ 9881.339141] LustreError: 390158:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9881.382680] LustreError: 390158:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9881.542224] Lustre: server umount lustre-OST0000 complete [ 9887.717230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9887.726346] LustreError: Skipped 1 previous similar message [ 9891.385380] LustreError: 390360:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 9891.388531] LustreError: 390360:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9891.436801] Lustre: server umount lustre-MDT0000 complete [ 9894.513557] LustreError: 386716:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761254927 with bad export cookie 586837523596184557 [ 9894.519984] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9894.528386] LustreError: 386716:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9900.927489] Lustre: server umount lustre-MDT0001 complete [ 9909.316759] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 9911.846840] Key type lgssc unregistered [ 9912.214620] LNet: 391092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9912.226254] LNetError: 391092:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9912.251579] LNet: Removed LNI 192.168.201.103@tcp [ 9913.007339] Key type .llcrypt unregistered [ 9913.009219] Key type ._llcrypt unregistered [ 9943.489939] Key type ._llcrypt registered [ 9943.492678] Key type .llcrypt registered [ 9943.594043] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 9958.020579] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 9959.331500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9959.382064] alg: No test for adler32 (adler32-zlib) [ 9960.373984] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [ 9960.632286] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 9962.360231] Key type lgssc registered [ 9963.802173] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9973.708408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9981.089288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9987.430521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9993.383047] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10001.035110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10001.094308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10001.124291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10002.363941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10002.405795] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10002.525189] Lustre: lustre-MDT0000: new disk, initializing [10002.613915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10002.627510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10006.426647] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10016.304939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10016.386644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10016.528525] Lustre: 395044:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10016.600808] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10016.604876] Lustre: Skipped 1 previous similar message [10016.738615] Lustre: lustre-MDT0001: new disk, initializing [10016.883679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10016.923873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10016.930809] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10017.065879] Lustre: 395842:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [10017.079705] Lustre: 395842:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [10017.091430] Lustre: 395842:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [10017.099762] Lustre: 395842:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [10017.106445] Lustre: 395842:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [10017.121570] Lustre: 395842:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [10020.852528] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10029.167532] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10032.996376] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10039.880028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10039.921680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10040.094701] Lustre: lustre-OST0000: new disk, initializing [10040.107553] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10040.190902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10046.056249] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10050.659436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10050.669097] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10050.816617] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10053.293241] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10057.310601] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10057.525738] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10061.164544] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10061.437972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10066.401383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10066.406135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10066.418494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10066.451508] Lustre: Skipped 1 previous similar message [10069.586958] LustreError: 397987:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10069.750596] Lustre: server umount lustre-OST0000 complete [10076.642101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10076.646756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10076.673170] Lustre: Skipped 1 previous similar message [10076.689988] Lustre: Skipped 1 previous similar message [10079.827913] LustreError: 398189:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10079.830900] LustreError: 398189:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10080.005161] Lustre: server umount lustre-MDT0000 complete [10084.491176] LustreError: 395038:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761255117 with bad export cookie 13227432138027182490 [10084.499805] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10084.516460] LustreError: 395038:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10084.678134] LustreError: 398390:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10084.690302] LustreError: 398390:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10085.000469] Lustre: server umount lustre-MDT0001 complete [10094.177643] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 17:32:06 (1761255126) [10103.091206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10110.487231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10110.956459] Lustre: lustre-MDT0000: new disk, initializing [10110.999739] LustreError: 399658:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10111.033169] LustreError: 399658:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [10111.092889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10114.482623] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10116.587706] LustreError: 399658:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10116.610283] LustreError: 399658:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [10120.684558] LustreError: 399658:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10122.968197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10123.408115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10127.245306] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10131.841830] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10135.173563] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10138.598070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10138.605569] LustreError: Skipped 1 previous similar message [10138.607511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10138.639275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10141.155941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10141.158408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10143.353496] LustreError: 401004:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10143.372830] LustreError: 401004:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10143.504067] Lustre: server umount lustre-MDT0000 complete [10146.277848] LustreError: 399657:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10146.295590] LustreError: 399657:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [10146.988272] LustreError: 399639:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761255180 with bad export cookie 13227432138027183414 [10146.998468] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10147.013631] LustreError: 399639:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [10147.471819] Lustre: server umount lustre-MDT0001 complete [10156.827268] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 17:33:09 (1761255189) [10163.642444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10164.113517] LustreError: 401925:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10164.186918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10167.477825] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10174.434872] LustreError: 401924:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10174.454959] LustreError: 401924:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [10175.681077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10180.614413] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10186.574206] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10190.083727] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10195.437032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10195.800508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10195.804827] Lustre: Skipped 1 previous similar message [10200.665389] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10205.929498] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10217.498216] Lustre: Permanently deactivating lustre-OST0000 [10217.540055] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10224.708889] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10227.088670] LustreError: 404570:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10227.103185] LustreError: 404570:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [10227.110187] LustreError: 404570:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10230.104603] LustreError: 404619:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [10230.116475] LustreError: 404619:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10238.220724] LustreError: 404716:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10238.235051] LustreError: 404716:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [10238.469745] Lustre: server umount lustre-OST0000 complete [10241.399619] LustreError: 404868:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [10241.403972] LustreError: 404868:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10245.093649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10245.101346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10245.113657] Lustre: Skipped 1 previous similar message [10245.116045] Lustre: Skipped 1 previous similar message [10255.329501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10255.339244] Lustre: Skipped 4 previous similar messages [10257.376370] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10257.551983] LustreError: 404965:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10257.566958] LustreError: 404965:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10257.664397] Lustre: server umount lustre-MDT0000 complete [10260.453219] LustreError: 401919:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10260.481936] LustreError: 401919:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [10261.297286] LustreError: 401904:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761255294 with bad export cookie 13227432138027183967 [10261.304891] LustreError: 401904:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [10261.308659] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10270.850927] Lustre: server umount lustre-MDT0001 complete [10278.372779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10281.373978] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10283.577381] LustreError: 406012:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10289.572824] Lustre: 406352:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.103@tcp [10291.165778] LustreError: 406449:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [10291.170578] LustreError: 406449:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [10291.265739] Lustre: server umount MGS complete [10300.171883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10300.491547] LustreError: 407077:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10300.506534] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10300.510725] Lustre: Skipped 1 previous similar message [10300.571623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10303.885242] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10310.017375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10310.213177] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10313.016229] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10316.932846] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10320.201972] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10322.075137] Lustre: Permanently reactivating lustre-OST0000 [10322.106677] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10322.112168] Lustre: Skipped 2 previous similar messages [10351.584210] LustreError: 408682:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10361.881220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10370.279591] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10375.553591] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10381.799551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10381.809623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10381.824577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10382.082193] LustreError: 410253:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10382.094378] LustreError: 410253:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [10382.227664] Lustre: server umount lustre-OST0000 complete [10386.914882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10396.204672] Lustre: server umount lustre-MDT0000 complete [10398.178242] LustreError: 407077:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10398.190300] LustreError: 407077:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [10398.974799] LustreError: 407058:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761255432 with bad export cookie 13227432138027184926 [10398.975391] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10398.980102] LustreError: 407058:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10399.293297] Lustre: server umount lustre-MDT0001 complete [10407.812578] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [10409.941605] Key type lgssc unregistered [10410.206845] LNet: 411187:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10410.211083] LNetError: 411187:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10410.224543] LNet: Removed LNI 192.168.201.103@tcp [10410.814794] Key type .llcrypt unregistered [10410.816433] Key type ._llcrypt unregistered [10424.100389] Key type ._llcrypt registered [10424.102855] Key type .llcrypt registered [10424.219178] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [10435.855688] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [10436.798835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10436.858854] alg: No test for adler32 (adler32-zlib) [10437.930799] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [10438.134711] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [10439.784175] Key type lgssc registered [10440.721142] Lustre: Echo OBD driver; http://www.lustre.org/ [10447.617532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10453.793285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10460.169550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10466.437292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10472.829524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10472.896347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10472.910715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10474.184347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10474.218866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10474.270791] Lustre: lustre-MDT0000: new disk, initializing [10474.383293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10474.415560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10478.514781] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10488.048797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10488.114857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10488.227795] Lustre: 414298:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10488.283431] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10488.286022] Lustre: Skipped 1 previous similar message [10488.355295] Lustre: lustre-MDT0001: new disk, initializing [10488.483034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10488.539657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10488.546476] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10491.698634] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10497.830472] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10500.629457] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10506.048824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10506.126324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10506.437982] Lustre: lustre-OST0000: new disk, initializing [10506.445760] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10506.545864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10511.322646] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10516.534068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10516.541382] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10516.640963] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10517.591746] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10521.243786] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10521.451303] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10524.994992] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10525.183641] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10527.203926] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10527.207540] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10527.251770] Lustre: Skipped 1 previous similar message [10527.256319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10532.331552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10532.345161] Lustre: Skipped 2 previous similar messages [10533.121649] LustreError: 417241:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10533.374869] Lustre: server umount lustre-OST0000 complete [10537.447034] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10537.454612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10537.467566] Lustre: Skipped 1 previous similar message [10542.563117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10542.571531] Lustre: Skipped 2 previous similar messages [10542.910336] LustreError: 417442:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10542.917896] LustreError: 417442:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10543.001153] Lustre: server umount lustre-MDT0000 complete [10546.439377] LustreError: 414291:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761255579 with bad export cookie 3037125172824080803 [10546.446668] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10546.455485] LustreError: 414291:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10546.622866] LustreError: 417644:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10546.626739] LustreError: 417644:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10546.961062] Lustre: server umount lustre-MDT0001 complete [10553.800727] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10555.070761] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 17:39:48 (1761255588) [10561.302059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10561.937060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10565.963859] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10567.146894] LustreError: 418364:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10567.187810] LustreError: 418364:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [10572.259230] LustreError: 418364:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10573.282243] LustreError: 418364:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10574.271853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10574.582232] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10577.815090] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10582.792277] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10585.746476] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10593.103522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10593.602249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10599.871744] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10606.036629] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10608.039964] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10610.374062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10618.339690] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10618.341786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10618.349786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10618.356826] Lustre: Skipped 1 previous similar message [10620.352921] LustreError: 420803:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10620.356029] LustreError: 420803:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10620.537941] Lustre: server umount lustre-OST0000 complete [10626.018296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10626.023190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10628.584145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10628.596821] Lustre: Skipped 3 previous similar messages [10638.304160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10638.386161] LustreError: 421007:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10638.390707] LustreError: 421007:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10638.513117] Lustre: server umount lustre-MDT0000 complete [10641.993729] LustreError: 418345:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761255675 with bad export cookie 3037125172824081727 [10641.994863] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10642.018722] LustreError: 418345:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [10642.475901] Lustre: server umount lustre-MDT0001 complete [10651.865903] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [10654.620510] Key type lgssc unregistered [10654.884962] LNet: 421741:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10654.895950] LNetError: 421741:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10655.917241] LNet: Removed LNI 192.168.201.103@tcp [10656.792667] Key type .llcrypt unregistered [10656.794433] Key type ._llcrypt unregistered [10676.496256] Key type ._llcrypt registered [10676.498166] Key type .llcrypt registered [10676.590096] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [10676.827518] Key type .llcrypt unregistered [10676.829443] Key type ._llcrypt unregistered [10692.319889] Key type ._llcrypt registered [10692.321652] Key type .llcrypt registered [10692.455825] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [10693.481297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10693.617394] alg: No test for adler32 (adler32-zlib) [10694.618395] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [10694.813369] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [10696.488136] Key type lgssc registered [10697.388135] Lustre: Echo OBD driver; http://www.lustre.org/ [10704.315360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10704.333784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10705.818628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10709.298294] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10711.013936] LustreError: 423913:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10711.020813] LustreError: 423913:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [10716.133192] LustreError: 423914:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10716.993662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10717.245287] LustreError: 423914:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10717.267708] LustreError: 423914:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [10717.288797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10722.292560] LustreError: 423910:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10722.312273] LustreError: 423910:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [10722.776256] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10727.396658] LustreError: 423910:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10727.408159] LustreError: 423910:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [10728.882668] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10731.541347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10735.779336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10736.341691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10740.870246] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10745.064364] Lustre: 423908:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [10745.077466] Lustre: 423908:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [10745.088665] Lustre: 423908:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [10745.100175] Lustre: 423908:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 7/153/0, punch: 0/0/0, quota 1/3/0 [10745.103414] Lustre: 423908:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [10745.109250] Lustre: 423908:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [10750.856153] LustreError: 425971:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [10751.141651] Lustre: server umount lustre-OST0000 complete [10756.579373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10756.583492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10756.599845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10757.100428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10757.131041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10757.146764] Lustre: Skipped 1 previous similar message [10762.211368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10762.216707] Lustre: Skipped 1 previous similar message [10765.294043] LustreError: 426172:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10765.300253] LustreError: 426172:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10765.416651] Lustre: server umount lustre-MDT0000 complete [10767.331521] LustreError: 425525:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10767.340193] LustreError: 425525:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [10768.360955] Lustre: 423227:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761255785/real 1761255785] req@00000000ee0f88ea x1846810486269696/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761255801 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10768.361835] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10768.375234] Lustre: 423227:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10768.762539] LustreError: 423895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761255802 with bad export cookie 146734406143651953 [10768.764310] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10768.782207] LustreError: 423895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10768.991390] LustreError: 426376:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10768.994208] LustreError: 426376:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10769.181580] Lustre: server umount lustre-MDT0001 complete [10778.797813] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [10781.406553] Key type lgssc unregistered [10781.726676] LNet: 426908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10781.739898] LNetError: 426908:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10781.767622] LNet: Removed LNI 192.168.201.103@tcp [10782.448189] Key type .llcrypt unregistered [10782.449995] Key type ._llcrypt unregistered [10802.978248] Key type ._llcrypt registered [10802.979889] Key type .llcrypt registered [10803.092451] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [10803.855851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10803.931094] alg: No test for adler32 (adler32-zlib) [10804.911280] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [10805.129257] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [10806.803075] Key type lgssc registered [10807.783761] Lustre: Echo OBD driver; http://www.lustre.org/ [10814.714300] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10814.735833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10830.240391] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10835.360230] LustreError: 428278:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.103@tcp: failed processing log, type 4: rc = -110 [10862.112316] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10866.702849] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10870.826415] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10877.505628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10878.330912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10878.439866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10882.185039] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10883.556297] LustreError: 428305:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10883.567919] LustreError: 428305:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [10888.682194] LustreError: 428305:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10889.427555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10889.703613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10892.815907] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10898.633142] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10903.199713] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10905.380904] Lustre: 430327:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 525, rollback = 2 [10905.386118] Lustre: 430327:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [10905.389949] Lustre: 430327:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 6/525/0 [10905.402177] Lustre: 430327:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 7/100/0, punch: 0/0/0, quota 1/3/0 [10905.409169] Lustre: 430327:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [10905.412955] Lustre: 430327:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [10910.187076] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10910.190778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10910.197789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10911.756661] LustreError: 430561:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [10912.030170] Lustre: server umount lustre-OST0000 complete [10916.840192] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10916.847682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10916.865171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10922.467330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10922.469943] Lustre: Skipped 2 previous similar messages [10925.556675] LustreError: 430762:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [10925.565629] LustreError: 430762:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10925.639209] Lustre: server umount lustre-MDT0000 complete [10927.585746] LustreError: 429204:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10927.617473] LustreError: 429204:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [10929.722687] LustreError: 428287:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761255963 with bad export cookie 5736740600471137196 [10929.730808] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10929.754381] LustreError: 428287:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10929.963214] LustreError: 430965:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10929.968823] LustreError: 430965:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10930.230753] Lustre: server umount lustre-MDT0001 complete [10940.469988] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [10942.592598] Key type lgssc unregistered [10942.855620] LNet: 431497:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10942.864824] LNetError: 431497:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10942.873595] LNet: Removed LNI 192.168.201.103@tcp [10943.609129] Key type .llcrypt unregistered [10943.610481] Key type ._llcrypt unregistered [10962.275489] Key type ._llcrypt registered [10962.276979] Key type .llcrypt registered [10962.425609] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [10963.295450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10963.306266] alg: No test for adler32 (adler32-zlib) [10964.457327] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [10964.663690] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [10966.321281] Key type lgssc registered [10967.343827] Lustre: Echo OBD driver; http://www.lustre.org/ [10974.158696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10974.212832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10975.819874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10979.927084] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10985.956000] LustreError: 432875:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10985.978645] LustreError: 432875:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [10988.418200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10988.790870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10992.117477] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10996.883832] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11000.170896] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11005.985126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11006.320936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11011.405250] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11016.027931] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11020.664401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11024.864990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11024.867904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11024.882355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11025.897875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11025.903217] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11025.915034] Lustre: Skipped 1 previous similar message [11027.245418] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.3@tcp (stopping) [11027.248490] Lustre: Skipped 2 previous similar messages [11030.075889] LustreError: 435123:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11030.186127] Lustre: server umount lustre-MDT0000 complete [11031.015401] LustreError: 432875:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11032.210276] LustreError: 432856:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.3@tcp arrived at 1761256065 with bad export cookie 9306519886513783704 [11032.223707] LustreError: 432856:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [11034.017459] LustreError: 432856:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761256067 with bad export cookie 9306519886513783039 [11034.027098] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11034.030046] LustreError: 432856:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [11035.113461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11035.117566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11035.135551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11036.129517] LustreError: 432876:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11036.159669] LustreError: 432876:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 3 previous similar messages [11039.142893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11039.143241] LustreError: 432871:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11039.144783] Lustre: Skipped 1 previous similar message [11039.157729] LustreError: 432871:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [11040.122416] LustreError: 435326:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11040.129531] LustreError: 435326:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11040.252176] Lustre: server umount lustre-OST0000 complete [11044.261921] LustreError: 432875:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11044.275702] LustreError: 432875:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [11045.515054] LustreError: 435680:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [11045.526843] LustreError: 435680:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11045.734338] Lustre: server umount lustre-MDT0001 complete [11054.033127] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [11056.108491] Key type lgssc unregistered [11056.309766] LNet: 436212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11056.317810] LNetError: 436212:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11057.383528] LNet: Removed LNI 192.168.201.103@tcp [11058.031609] Key type .llcrypt unregistered [11058.034078] Key type ._llcrypt unregistered [11079.098696] Key type ._llcrypt registered [11079.100017] Key type .llcrypt registered [11079.258854] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [11080.293560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11080.310067] alg: No test for adler32 (adler32-zlib) [11081.312441] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [11081.568657] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [11083.216171] Key type lgssc registered [11084.210412] Lustre: Echo OBD driver; http://www.lustre.org/ [11091.600190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11091.628342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11093.025717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11097.653449] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11103.208185] LustreError: 437613:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11103.219791] LustreError: 437613:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [11107.436432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11107.826222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11112.042882] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11117.253992] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11120.985277] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11127.880121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11128.252921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11133.498117] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11137.860438] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11138.487943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11143.649731] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11143.658736] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11143.680355] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11143.720910] Lustre: Skipped 1 previous similar message [11145.527467] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.3@tcp (stopping) [11145.532438] Lustre: Skipped 1 previous similar message [11148.033039] LustreError: 439862:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [11148.112527] Lustre: server umount lustre-MDT0001 complete [11148.772853] LustreError: 437608:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11148.795220] LustreError: 437608:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [11153.893365] LustreError: 438327:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11153.925828] LustreError: 438327:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [11159.012488] LustreError: 438327:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11159.012746] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11159.016837] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11159.074762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11160.318701] LustreError: 440065:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11160.323465] LustreError: 440065:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11160.507765] Lustre: server umount lustre-OST0000 complete [11164.046383] LustreError: 440266:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11164.049758] LustreError: 440266:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11164.244130] Lustre: server umount lustre-MDT0000 complete [11175.937972] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [11178.091848] Key type lgssc unregistered [11178.339083] LNet: 440950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11178.348644] LNetError: 440950:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11178.359413] LNet: Removed LNI 192.168.201.103@tcp [11178.997510] Key type .llcrypt unregistered [11178.999096] Key type ._llcrypt unregistered [11199.499420] Key type ._llcrypt registered [11199.501490] Key type .llcrypt registered [11199.598943] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [11210.351507] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [11210.879298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11210.898458] alg: No test for adler32 (adler32-zlib) [11211.925404] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [11212.074303] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [11213.712208] Key type lgssc registered [11214.731756] Lustre: Echo OBD driver; http://www.lustre.org/ [11221.166600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11225.625831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11230.463833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11236.028316] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11243.332778] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [11251.165936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11256.563115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11256.615933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11256.632761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11257.842186] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11257.880262] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11257.983946] Lustre: lustre-MDT0000: new disk, initializing [11258.099664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11258.113468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11260.905422] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11267.656315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11272.908815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11272.985715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11273.173388] Lustre: lustre-OST0000: new disk, initializing [11273.176552] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11273.179420] Lustre: Skipped 1 previous similar message [11273.236270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11274.723240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11274.731777] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11274.787247] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11278.462390] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11285.363893] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11293.267290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11298.873776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11298.918979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11299.022614] Lustre: 444904:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11299.092852] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11299.156322] Lustre: lustre-MDT0001: new disk, initializing [11299.270538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11299.316450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11299.322726] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11299.396577] Lustre: 447147:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [11299.404389] Lustre: 447147:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [11299.410510] Lustre: 447147:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [11299.424136] Lustre: 447147:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [11299.429451] Lustre: 447147:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [11299.437321] Lustre: 447147:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [11302.405913] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11310.691952] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11310.980250] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11315.151211] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11315.385519] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11321.653535] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.3@tcp (stopping) [11322.853053] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11322.862103] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11322.873169] Lustre: Skipped 1 previous similar message [11322.879211] Lustre: Skipped 1 previous similar message [11326.470633] LustreError: 448328:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11326.575319] Lustre: server umount lustre-MDT0001 complete [11333.092405] LustreError: 444911:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11333.107441] LustreError: 444911:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 5 previous similar messages [11338.210443] LustreError: 444911:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11338.228392] LustreError: 444911:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 1 previous similar message [11343.332254] LustreError: 444917:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11343.332575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11343.362143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11348.455046] LustreError: 444917:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11348.461382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11349.374153] LustreError: 448601:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [11349.378977] LustreError: 448601:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11349.571421] Lustre: server umount lustre-OST0000 complete [11367.392504] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11367.515246] LustreError: 448802:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [11367.532226] LustreError: 448802:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11367.856080] Lustre: server umount lustre-MDT0000 complete [11379.865029] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [11382.285304] Key type lgssc unregistered [11382.552935] LNet: 449487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11382.557243] LNetError: 449487:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11382.573981] LNet: Removed LNI 192.168.201.103@tcp [11383.342563] Key type .llcrypt unregistered [11383.345549] Key type ._llcrypt unregistered [11414.656384] Key type ._llcrypt registered [11414.659876] Key type .llcrypt registered [11414.728299] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [11414.901804] Key type .llcrypt unregistered [11414.903601] Key type ._llcrypt unregistered [11429.251698] Key type ._llcrypt registered [11429.253436] Key type .llcrypt registered [11429.365705] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [11430.188420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11430.256111] alg: No test for adler32 (adler32-zlib) [11431.484846] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [11431.938285] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [11433.712806] Key type lgssc registered [11434.811752] Lustre: Echo OBD driver; http://www.lustre.org/ [11442.322097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11448.623283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11454.500973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11461.619328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11461.685321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11461.708158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11463.033390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11463.085496] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11463.157698] Lustre: lustre-MDT0000: new disk, initializing [11463.264949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11463.301138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11467.680664] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11477.351641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11477.427702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11477.545149] Lustre: 452835:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11477.619844] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11477.622738] Lustre: Skipped 1 previous similar message [11477.718486] Lustre: lustre-MDT0001: new disk, initializing [11477.828160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11477.885400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11477.900937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11477.985368] Lustre: 453632:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [11478.008776] Lustre: 453632:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [11478.030422] Lustre: 453632:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [11478.042920] Lustre: 453632:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [11478.056883] Lustre: 453632:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [11478.070997] Lustre: 453632:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [11481.474374] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11488.121828] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11491.293418] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11498.043425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11498.124714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11498.557546] Lustre: lustre-OST0000: new disk, initializing [11498.568940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11498.672675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11504.191882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11504.196574] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11504.308249] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11504.757106] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11512.391262] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11518.952482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11518.974926] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11519.000033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11519.971602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11519.971632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11524.021324] LustreError: 455276:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11524.119070] Lustre: server umount lustre-MDT0000 complete [11527.398959] LustreError: 452826:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761256560 with bad export cookie 1909747908142638392 [11527.403960] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11527.430237] LustreError: 452826:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11527.565516] LustreError: 455478:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11527.612857] LustreError: 455478:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11527.857031] Lustre: server umount lustre-MDT0001 complete [11540.884345] LustreError: 455678:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [11540.908753] LustreError: 455678:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11541.129325] Lustre: server umount lustre-OST0000 complete [11551.371703] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 17:56:23 (1761256583) [11552.833817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11561.147922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11561.660513] LustreError: 456508:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11561.693155] LustreError: 456508:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 4 previous similar messages [11561.821610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11566.832353] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11567.082378] LustreError: 456508:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11572.203768] LustreError: 456508:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11576.828281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11577.390711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11581.875581] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11587.489363] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11591.108627] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11597.889173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11597.990880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11598.230186] Lustre: Found index 0 for lustre-OST0000, updating log [11598.477848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11604.227571] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11607.672076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [11609.980707] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11618.279598] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11618.282441] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11618.283720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11618.283726] Lustre: Skipped 2 previous similar messages [11618.295117] LustreError: Skipped 1 previous similar message [11618.305133] Lustre: Skipped 2 previous similar messages [11619.359216] LustreError: 458768:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11619.370063] LustreError: 458113:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11619.373669] LustreError: 458768:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11619.527148] Lustre: server umount lustre-OST0000 complete [11622.134362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11624.941667] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11624.977458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11624.994324] Lustre: Skipped 2 previous similar messages [11630.254572] LustreError: 459023:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11630.257764] LustreError: 459023:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11630.336926] Lustre: server umount lustre-MDT0000 complete [11631.082176] LustreError: 457222:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11631.104459] LustreError: 457222:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [11633.611267] LustreError: 457229:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761256666 with bad export cookie 1909747908142639827 [11633.617541] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11633.624624] LustreError: 457229:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [11634.024698] Lustre: server umount lustre-MDT0001 complete [11642.085265] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 17:57:54 (1761256674) [11643.074865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11644.146544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11645.002863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11645.950128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11646.957201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11647.886085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11648.812609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11649.702314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11650.788250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11652.100443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11653.312148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11654.471937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11655.790799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11656.874833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11657.927552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11659.110904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11660.325942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11661.421857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11662.553704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11663.536049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11664.649298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11665.804208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11667.023927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11668.123510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11669.257172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11670.332565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11671.271558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11672.265334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11673.142686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11674.134736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11675.119269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11676.044227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11676.974436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11677.991040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11678.926909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11679.884949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11680.914511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11681.808339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11682.825574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11683.853266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11684.852752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11685.959220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11687.057233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11687.944579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11688.849333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11689.848175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11690.911470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11691.980120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11692.976073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11693.994090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11694.889245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11702.969869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11703.569378] LustreError: 462623:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11703.671859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11708.396447] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11718.014730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11723.114994] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11729.547239] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11733.530234] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11739.736386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11739.845970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11740.057138] Lustre: Found index 0 for lustre-OST0000, updating log [11740.328405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11740.332596] Lustre: Skipped 1 previous similar message [11745.607710] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11750.353905] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11750.577851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [11756.003329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11756.007858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11756.030332] Lustre: Skipped 1 previous similar message [11756.037517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11756.063509] Lustre: Skipped 3 previous similar messages [11758.150090] LustreError: 464892:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11758.166122] LustreError: 464892:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [11758.320450] Lustre: server umount lustre-OST0000 complete [11764.194196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11764.209436] Lustre: Skipped 5 previous similar messages [11768.518946] Lustre: server umount lustre-MDT0000 complete [11769.320501] LustreError: 462622:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11769.343784] LustreError: 462622:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 4 previous similar messages [11772.311353] LustreError: 462603:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761256805 with bad export cookie 1909747908142640905 [11772.321196] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11772.323910] LustreError: 462603:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [11772.802495] Lustre: server umount lustre-MDT0001 complete [11775.428935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11790.154436] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [11799.361679] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [11809.517797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11816.898197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11823.387762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11829.984662] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11837.409402] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 18:01:09 (1761256869) [11852.749690] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [11855.250874] Key type lgssc unregistered [11855.517880] LNet: 468875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11855.522094] LNetError: 468875:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11876.008808] LNet: Removed LNI 192.168.201.103@tcp [11877.006343] Key type .llcrypt unregistered [11877.012796] Key type ._llcrypt unregistered [11895.017204] Key type ._llcrypt registered [11895.018590] Key type .llcrypt registered [11895.213101] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [11896.007822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11896.094226] alg: No test for adler32 (adler32-zlib) [11897.378876] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [11897.635890] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [11899.328251] Key type lgssc registered [11900.514930] Lustre: Echo OBD driver; http://www.lustre.org/ [11905.120456] LNet: Added LNI 192.168.201.103@tcp100 [8/256/0/180] [11906.990755] Lustre: DEBUG MARKER: SKIP: conf-sanity test_73c unable to configure net [11926.562204] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [11930.119691] Key type lgssc unregistered [11930.641944] LNet: 470750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11930.658402] LNetError: 470750:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11930.668351] LNet: Removed LNI 192.168.201.103@tcp [11930.679344] LNet: Removed LNI 192.168.201.103@tcp100 [11931.479702] Key type .llcrypt unregistered [11931.481412] Key type ._llcrypt unregistered [11958.484305] Key type ._llcrypt registered [11958.486187] Key type .llcrypt registered [11958.630880] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [11974.534965] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [11975.818174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11975.883068] alg: No test for adler32 (adler32-zlib) [11976.952362] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [11977.198758] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [11978.888222] Key type lgssc registered [11980.374891] Lustre: Echo OBD driver; http://www.lustre.org/ [11990.008318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11995.507038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12001.333035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12007.744869] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12013.473762] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 18:04:06 (1761257046) [12029.552807] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [12032.628879] Key type lgssc unregistered [12033.006947] LNet: 474801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12033.023278] LNetError: 474801:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12033.044326] LNet: Removed LNI 192.168.201.103@tcp [12033.891799] Key type .llcrypt unregistered [12033.893768] Key type ._llcrypt unregistered [12051.057634] Key type ._llcrypt registered [12051.059279] Key type .llcrypt registered [12051.149767] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [12052.309489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12052.389700] alg: No test for adler32 (adler32-zlib) [12053.463226] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [12053.678364] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [12055.360442] Key type lgssc registered [12056.600056] Lustre: Echo OBD driver; http://www.lustre.org/ [12067.484222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12067.544201] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12067.569545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12068.968368] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12069.008229] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12069.096848] Lustre: lustre-MDT0000: new disk, initializing [12069.248254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12069.276450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12073.117510] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12084.787164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12084.867842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12084.931307] Lustre: 475999:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12085.006338] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12085.009631] Lustre: Skipped 1 previous similar message [12085.089454] Lustre: lustre-MDT0001: new disk, initializing [12085.221348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12085.258595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12085.265923] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12085.330111] Lustre: 476795:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [12085.333781] Lustre: 476795:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [12085.392466] Lustre: 476795:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [12085.396693] Lustre: 476795:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [12085.419759] Lustre: 476795:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [12085.439696] Lustre: 476795:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [12088.902552] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12095.038580] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12098.133297] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12104.940880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12105.015726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12105.292213] Lustre: lustre-OST0000: new disk, initializing [12105.300417] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12105.398758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12110.868334] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12113.454509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12113.460559] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12113.560285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12117.902316] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12120.901830] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing lnet_if_list [12125.571580] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [12125.596106] LNet: Added LNI 192.168.201.103@tcp42 [8/256/0/180] [12139.491670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12139.496066] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12139.508833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12139.513578] Lustre: Skipped 1 previous similar message [12144.614471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12144.625466] Lustre: Skipped 1 previous similar message [12145.461368] LustreError: 479385:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [12145.621894] Lustre: server umount lustre-OST0000 complete [12149.738935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12149.745167] Lustre: Skipped 2 previous similar messages [12149.745276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12149.785475] Lustre: Skipped 1 previous similar message [12154.824202] LustreError: 479586:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [12154.827576] LustreError: 479586:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12154.851410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12154.862344] Lustre: Skipped 1 previous similar message [12154.992989] Lustre: server umount lustre-MDT0000 complete [12158.818240] LustreError: 475991:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761257192 with bad export cookie 3864058501220677801 [12158.820264] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12158.841934] LustreError: 475991:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12159.031954] LustreError: 479788:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12159.040360] LustreError: 479788:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12159.317273] Lustre: server umount lustre-MDT0001 complete [12168.626055] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [12171.781543] Key type lgssc unregistered [12172.188546] LNet: 480328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12172.196350] LNetError: 480328:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12173.216235] LNet: Removed LNI 192.168.201.103@tcp [12173.226523] LNetError: 480328:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12173.252189] LNet: Removed LNI 192.168.201.103@tcp42 [12174.174490] Key type .llcrypt unregistered [12174.179901] Key type ._llcrypt unregistered [12195.652426] Key type ._llcrypt registered [12195.657752] Key type .llcrypt registered [12195.798379] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [12196.932201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12197.179856] alg: No test for adler32 (adler32-zlib) [12198.621817] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [12198.983855] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [12200.688224] Key type lgssc registered [12202.157971] Lustre: Echo OBD driver; http://www.lustre.org/ [12211.336656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12211.383764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12213.200274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12217.114897] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12223.476549] LustreError: 481569:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12223.492330] LustreError: 481569:0:(ldlm_lib.c:1245:target_handle_connect()) Skipped 2 previous similar messages [12224.484637] LustreError: 481569:0:(ldlm_lib.c:1245:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12226.156876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12226.460533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12230.653391] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12236.610797] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12240.761231] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12248.418970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12248.833531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12254.543197] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12260.194283] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12269.439438] loop0: detected capacity change from 0 to 204800000 [12269.614983] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12269.626368] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12269.634377] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12269.684514] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12275.194270] loop0: detected capacity change from 0 to 204800000 [12275.399297] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12275.410534] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12275.415251] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12275.554771] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12281.313253] loop0: detected capacity change from 0 to 204800000 [12287.688577] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12287.764681] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12287.941969] Lustre: fs15246-OST0000: new disk, initializing [12287.948494] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [12288.051279] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12294.021290] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12304.456140] loop1: detected capacity change from 0 to 204800000 [12310.845446] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12310.947485] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12311.116535] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [12311.235705] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [12311.360879] Lustre: fs15246-MDT0000: new disk, initializing [12311.632906] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12311.662039] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12311.767577] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12312.803068] Lustre: 485952:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@000000008b9e88b8 x1846812063350656/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1761257362 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [12316.062948] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12317.863115] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12317.876234] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12317.934190] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12341.243158] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12341.254503] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [12344.155272] LustreError: 487077:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [12344.277823] Lustre: server umount fs15246-MDT0000 complete [12346.346525] LustreError: 481563:0:(ldlm_lib.c:1245:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12351.470831] LustreError: 481564:0:(ldlm_lib.c:1245:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12352.273322] LustreError: 487580:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [12352.277156] LustreError: 487580:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12352.316154] Lustre: server umount fs15246-OST0000 complete [12363.746655] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12363.753946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12363.773535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12366.824502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12366.832819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12366.846909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12368.445107] LustreError: 488082:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12368.460162] LustreError: 488082:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [12368.644123] Lustre: server umount lustre-OST0000 complete [12375.013430] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12375.027158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12375.041560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12375.045916] Lustre: Skipped 1 previous similar message [12378.359731] LustreError: 488285:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12378.363255] LustreError: 488285:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12378.459838] Lustre: server umount lustre-MDT0000 complete [12382.097032] LustreError: 481548:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761257415 with bad export cookie 4037034222158564448 [12382.121523] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12382.559141] Lustre: server umount lustre-MDT0001 complete [12393.920450] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [12396.941550] Key type lgssc unregistered [12397.290625] LNet: 489019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12397.299231] LNetError: 489019:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12398.315458] LNet: Removed LNI 192.168.201.103@tcp [12399.244803] Key type .llcrypt unregistered [12399.246887] Key type ._llcrypt unregistered [12417.759545] Key type ._llcrypt registered [12417.760913] Key type .llcrypt registered [12417.887622] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [12418.899456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12418.981060] alg: No test for adler32 (adler32-zlib) [12420.055187] Lustre: Lustre: Build Version: 2.16.59_45_g8a89fa6 [12420.307914] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [12422.000230] Key type lgssc registered [12423.059756] Lustre: Echo OBD driver; http://www.lustre.org/ [12429.345106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12433.554817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12438.308619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12442.737962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12458.181735] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [12467.850555] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [12476.441938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12482.428760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12487.714902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12493.147024] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12499.119693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12499.214674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12499.245101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12500.465192] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12500.496823] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12500.556608] Lustre: lustre-MDT0000: new disk, initializing [12500.640257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12500.677094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12503.674312] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12512.970863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12513.026520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12513.118489] Lustre: 493452:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12513.175581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12513.178262] Lustre: Skipped 1 previous similar message [12513.248798] Lustre: lustre-MDT0001: new disk, initializing [12513.353284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12513.406431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12513.421793] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12513.550664] Lustre: 494248:0:(osd_internal.h:1361:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 507 < left 524, rollback = 2 [12513.568974] Lustre: 494248:0:(osd_handler.c:2088:osd_trans_dump_creds()) create: 3/12/6, destroy: 0/0/0 [12513.573375] Lustre: 494248:0:(osd_handler.c:2095:osd_trans_dump_creds()) attr_set: 3/3/0, xattr_set: 5/524/0 [12513.578961] Lustre: 494248:0:(osd_handler.c:2105:osd_trans_dump_creds()) write: 6/142/0, punch: 0/0/0, quota 1/3/0 [12513.581979] Lustre: 494248:0:(osd_handler.c:2112:osd_trans_dump_creds()) insert: 7/135/3, delete: 0/0/0 [12513.594369] Lustre: 494248:0:(osd_handler.c:2119:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 0/0/0 [12517.440329] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12524.059608] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12527.406389] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12533.265700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12533.325291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12533.549520] Lustre: lustre-OST0000: new disk, initializing [12533.555173] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12533.627224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12538.984951] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12539.973981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12539.991601] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12540.137230] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12546.429971] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12550.045787] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12550.268821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12554.019919] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12554.232279] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12560.865366] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12560.868757] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12560.872748] LustreError: Skipped 1 previous similar message [12560.885103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12560.899149] Lustre: Skipped 1 previous similar message [12562.113955] LustreError: 496392:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [12562.231926] Lustre: server umount lustre-OST0000 complete [12565.985599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12565.994855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12565.996762] Lustre: Skipped 1 previous similar message [12565.999494] Lustre: Skipped 1 previous similar message [12571.111923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12571.117946] Lustre: Skipped 1 previous similar message [12571.235851] LustreError: 496593:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [12571.240114] LustreError: 496593:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12571.341105] Lustre: server umount lustre-MDT0000 complete [12574.611665] LustreError: 493444:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761257607 with bad export cookie 9502726819879464729 [12574.616319] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12574.627910] LustreError: 493444:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [12574.758118] LustreError: 496795:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12574.766318] LustreError: 496795:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12574.992138] Lustre: server umount lustre-MDT0001 complete [12594.926105] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [12604.238448] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [12612.632715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12617.806385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12623.128132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12630.056670] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12631.983190] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12388 sec ========= 18:14:24 (1761257664) [12633.703635] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:14:26 (1761257666) === [12636.557192] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:14:29 (1761257669) === [12670.203954] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [12672.639468] Key type lgssc unregistered [12672.879820] LNet: 501654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12672.891486] LNetError: 501654:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12672.907153] LNet: Removed LNI 192.168.201.103@tcp [12673.657909] Key type .llcrypt unregistered [12673.660268] Key type ._llcrypt unregistered