[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 485767294 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002319] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004007] kvm-guest: setup PV IPIs [ 0.006244] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007015] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008005] pid_max: default: 32768 minimum: 301 [ 0.009084] LSM: Security Framework initializing [ 0.009958] Yama: becoming mindful. [ 0.010019] SELinux: Initializing. [ 0.011045] *** VALIDATE selinux *** [ 0.017288] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.021629] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.022097] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.023062] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024060] *** VALIDATE tmpfs *** [ 0.025315] *** VALIDATE proc *** [ 0.026130] *** VALIDATE cgroup *** [ 0.026795] *** VALIDATE cgroup2 *** [ 0.027182] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.028089] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.029002] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.030015] Spectre V2 : User space: Vulnerable [ 0.031003] Speculative Store Bypass: Vulnerable [ 0.033665] debug: unmapping init [mem 0xffffffffb8e59000-0xffffffffb8e60fff] [ 0.035963] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.036430] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.037009] ... version: 2 [ 0.038005] ... bit width: 48 [ 0.038650] ... generic registers: 4 [ 0.039004] ... value mask: 0000ffffffffffff [ 0.039933] ... max period: 00007fffffffffff [ 0.040005] ... fixed-purpose events: 3 [ 0.040998] ... event mask: 000000070000000f [ 0.042199] rcu: Hierarchical SRCU implementation. [ 0.044105] smp: Bringing up secondary CPUs ... [ 0.045537] x86: Booting SMP configuration: [ 0.046026] .... node #0, CPUs: #1 #2 #3 [ 0.048381] smp: Brought up 1 node, 4 CPUs [ 0.049725] smpboot: Max logical packages: 1 [ 0.050005] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.266168] node 0 deferred pages initialised in 215ms [ 0.268129] devtmpfs: initialized [ 0.269195] x86/mm: Memory block size: 128MB [ 0.271316] gcov: version magic: 0x41383552 [ 0.273015] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.274065] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.275228] pinctrl core: initialized pinctrl subsystem [ 0.276105] [ 0.276389] ************************************************************* [ 0.277006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.278006] ** ** [ 0.279006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.280007] ** ** [ 0.281008] ** This means that this kernel is built to expose internal ** [ 0.282005] ** IOMMU data structures, which may compromise security on ** [ 0.283005] ** your system. ** [ 0.284005] ** ** [ 0.285006] ** If you see this message and you are not debugging the ** [ 0.286005] ** kernel, report this immediately to your vendor! ** [ 0.287005] ** ** [ 0.288005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.289007] ************************************************************* [ 0.290530] NET: Registered protocol family 16 [ 0.291280] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.292023] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.293028] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.294331] cpuidle: using governor menu [ 0.295547] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.296346] PCI: Using configuration type 1 for base access [ 0.297082] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.302154] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.305032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.308122] cryptd: max_cpu_qlen set to 1000 [ 0.310177] ACPI: Added _OSI(Module Device) [ 0.311011] ACPI: Added _OSI(Processor Device) [ 0.313013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.314008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.317719] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.325045] ACPI: Interpreter enabled [ 0.326121] ACPI: PM: (supports S0 S3 S4 S5) [ 0.327012] ACPI: Using IOAPIC for interrupt routing [ 0.329233] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.334484] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.344224] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.346021] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.348009] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.350054] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.355046] acpiphp: Slot [2] registered [ 0.356099] acpiphp: Slot [3] registered [ 0.357065] acpiphp: Slot [4] registered [ 0.358077] acpiphp: Slot [5] registered [ 0.360085] acpiphp: Slot [6] registered [ 0.361104] acpiphp: Slot [7] registered [ 0.362071] acpiphp: Slot [8] registered [ 0.363106] acpiphp: Slot [9] registered [ 0.365082] acpiphp: Slot [10] registered [ 0.366075] acpiphp: Slot [11] registered [ 0.367060] acpiphp: Slot [12] registered [ 0.368101] acpiphp: Slot [13] registered [ 0.370063] acpiphp: Slot [14] registered [ 0.372084] acpiphp: Slot [15] registered [ 0.373049] acpiphp: Slot [16] registered [ 0.374077] acpiphp: Slot [17] registered [ 0.374895] acpiphp: Slot [18] registered [ 0.376058] acpiphp: Slot [19] registered [ 0.377056] acpiphp: Slot [20] registered [ 0.379077] acpiphp: Slot [21] registered [ 0.380044] acpiphp: Slot [22] registered [ 0.381060] acpiphp: Slot [23] registered [ 0.382049] acpiphp: Slot [24] registered [ 0.383090] acpiphp: Slot [25] registered [ 0.385054] acpiphp: Slot [26] registered [ 0.386055] acpiphp: Slot [27] registered [ 0.387062] acpiphp: Slot [28] registered [ 0.388060] acpiphp: Slot [29] registered [ 0.390118] acpiphp: Slot [30] registered [ 0.391059] acpiphp: Slot [31] registered [ 0.392045] PCI host bridge to bus 0000:00 [ 0.393009] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.395009] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.396021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.399022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.401019] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.403029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.404173] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.406860] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.409458] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.414860] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.418060] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.422051] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.425017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.426018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.428388] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.430718] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.432040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.434655] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.437836] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.448018] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.452023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.458971] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.468019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.475015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.490025] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.502925] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.575031] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.639076] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.657020] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.665698] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.674022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.680017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.700012] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.708108] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.714013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.718020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.732020] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.741037] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.747013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.754018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.767015] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.777489] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.782012] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.787015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.807018] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.819213] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.821292] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.823279] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.824275] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.826200] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.830194] iommu: Default domain type: Passthrough [ 0.831313] SCSI subsystem initialized [ 0.833073] ACPI: bus type USB registered [ 0.833877] usbcore: registered new interface driver usbfs [ 0.835036] usbcore: registered new interface driver hub [ 0.836033] usbcore: registered new device driver usb [ 0.837080] pps_core: LinuxPPS API ver. 1 registered [ 0.839010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.840029] PTP clock support registered [ 0.842233] EDAC MC: Ver: 3.0.0 [ 0.843507] PCI: Using ACPI for IRQ routing [ 0.844864] NetLabel: Initializing [ 0.845008] NetLabel: domain hash size = 128 [ 0.846006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.847144] NetLabel: unlabeled traffic allowed by default [ 0.849048] vgaarb: loaded [ 0.850216] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.851005] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.855343] clocksource: Switched to clocksource kvm-clock [ 0.950174] VFS: Disk quotas dquot_6.6.0 [ 0.951660] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.953914] *** VALIDATE ramfs *** [ 0.954548] *** VALIDATE hugetlbfs *** [ 0.955599] pnp: PnP ACPI init [ 0.957413] pnp: PnP ACPI: found 6 devices [ 0.974852] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.977710] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.979890] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.982301] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.984830] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.986677] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.989598] NET: Registered protocol family 2 [ 0.992247] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.996311] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.000018] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.005782] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.009131] TCP: Hash tables configured (established 65536 bind 65536) [ 1.011863] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.014344] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.016257] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.018221] NET: Registered protocol family 1 [ 1.020207] RPC: Registered named UNIX socket transport module. [ 1.021964] RPC: Registered udp transport module. [ 1.023135] RPC: Registered tcp transport module. [ 1.023965] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.025647] NET: Registered protocol family 44 [ 1.027508] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.029346] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.031037] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.032575] PCI: CLS 0 bytes, default 64 [ 1.034126] Unpacking initramfs... [ 2.496259] debug: unmapping init [mem 0xffff99e0fcc54000-0xffff99e0fffbffff] [ 2.500339] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.502295] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.504774] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.124377] Initialise system trusted keyrings [ 3.125789] Key type blacklist registered [ 3.128866] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.137694] zbud: loaded [ 3.140368] *** VALIDATE nfs *** [ 3.141615] *** VALIDATE nfs4 *** [ 3.143142] pstore: using deflate compression [ 3.146950] Platform Keyring initialized [ 3.295803] NET: Registered protocol family 38 [ 3.297591] Key type asymmetric registered [ 3.298964] Asymmetric key parser 'x509' registered [ 3.300421] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.302862] io scheduler mq-deadline registered [ 3.304199] io scheduler kyber registered [ 3.305533] io scheduler bfq registered [ 3.307076] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.309354] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.311595] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.313826] ACPI: Power Button [PWRF] [ 3.408928] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.516477] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.717973] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.812830] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.020275] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.054363] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.088152] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.093209] Non-volatile memory driver v1.3 [ 4.094561] Linux agpgart interface v0.103 [ 4.131748] virtio_blk virtio1: [vda] 132808 512-byte logical blocks (68.0 MB/64.8 MiB) [ 4.135065] vda: detected capacity change from 0 to 67997696 [ 4.152370] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.157833] vdb: detected capacity change from 0 to 1073741824 [ 4.176353] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.179081] vdc: detected capacity change from 0 to 2621440000 [ 4.197984] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.201172] vdd: detected capacity change from 0 to 2621440000 [ 4.219985] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.224350] vde: detected capacity change from 0 to 4294967296 [ 4.254934] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.257919] vdf: detected capacity change from 0 to 4294967296 [ 4.266413] libphy: Fixed MDIO Bus: probed [ 4.274905] usbcore: registered new interface driver usbserial_generic [ 4.276812] usbserial: USB Serial support registered for generic [ 4.279396] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.284657] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.286698] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.288834] mousedev: PS/2 mouse device common for all mice [ 4.291414] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.293272] rtc_cmos 00:05: RTC can wake from S4 [ 4.301676] rtc_cmos 00:05: registered as rtc0 [ 4.303544] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.305580] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.310305] intel_pstate: CPU model not supported [ 4.316645] hid: raw HID events driver (C) Jiri Kosina [ 4.339949] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.345545] usbcore: registered new interface driver usbhid [ 4.348179] usbhid: USB HID core driver [ 4.349963] drop_monitor: Initializing network drop monitor service [ 4.352334] Initializing XFRM netlink socket [ 4.353898] NET: Registered protocol family 10 [ 4.358138] Segment Routing with IPv6 [ 4.359877] NET: Registered protocol family 17 [ 4.362323] mpls_gso: MPLS GSO support [ 4.368940] RAS: Correctable Errors collector initialized. [ 4.370931] AVX version of gcm_enc/dec engaged. [ 4.372378] AES CTR mode by8 optimization enabled [ 4.496770] sched_clock: Marking stable (4496752983, 0)->(5301491453, -804738470) [ 4.503526] registered taskstats version 1 [ 4.506528] Loading compiled-in X.509 certificates [ 4.508333] zswap: loaded using pool lzo/zbud [ 4.555775] Key type big_key registered [ 4.578431] Key type encrypted registered [ 4.580385] ima: No TPM chip found, activating TPM-bypass! [ 4.583371] ima: Allocated hash algorithm: sha1 [ 4.585036] ima: No architecture policies found [ 4.586376] evm: Initialising EVM extended attributes: [ 4.588582] evm: security.selinux [ 4.589941] evm: security.ima [ 4.591308] evm: security.capability [ 4.592611] evm: HMAC attrs: 0x1 [ 4.602042] rtc_cmos 00:05: setting system clock to 2025-08-19 20:54:29 UTC (1755636869) [ 4.608866] debug: unmapping init [mem 0xffffffffb9e03000-0xffffffffb9ffffff] [ 4.611696] debug: unmapping init [mem 0xffffffffb8b82000-0xffffffffb8e58fff] [ 4.620151] Write protecting the kernel read-only data: 28672k [ 4.626401] debug: unmapping init [mem 0xffffffffb7203000-0xffffffffb73fffff] [ 4.629800] debug: unmapping init [mem 0xffffffffb7b14000-0xffffffffb7bfffff] [ 4.683554] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.698346] systemd[1]: Detected virtualization kvm. [ 4.700715] systemd[1]: Detected architecture x86-64. [ 4.706856] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.730152] systemd[1]: No hostname configured. [ 4.733855] systemd[1]: Set hostname to . [ 4.736257] random: systemd: uninitialized urandom read (16 bytes read) [ 4.738986] systemd[1]: Initializing machine ID from random generator. [ 4.793573] random: ln: uninitialized urandom read (6 bytes read) [ 4.980426] random: systemd: uninitialized urandom read (16 bytes read) [ 4.983272] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.993386] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 4.998532] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.014429] device-mapper: uevent: version 1.0.3 [ 6.015987] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.927241] virtio_net virtio0 ens2: renamed from eth0 [ 6.976260] random: fast init done [ 7.331813] scsi host0: ata_piix [ 7.347444] scsi host1: ata_piix [ 7.360794] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.362910] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.110939] random: crng init done [ 12.112136] random: 7 urandom warning(s) missed due to ratelimiting [ 13.615147] dracut-initqueue[588]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 14.287754] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.577275] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.088335] SELinux: Disabled at runtime. [ 16.158174] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.165767] systemd[1]: Detected virtualization kvm. [ 16.167622] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.081326] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.088685] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.099848] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.109282] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.116238] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.140110] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.156256] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. [ 17.414480] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 18.084133] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.672132] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.722341] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.076471] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.101106] EDAC sbridge: Ver: 1.1.2 [ 21.433353] Key type dns_resolver registered [ 21.796824] NFS: Registering the id_resolver key type [ 21.798793] Key type id_resolver registered [ 21.800785] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg432-server login: [ 44.581041] libcfs: loading out-of-tree module taints kernel. [ 44.588128] Key type ._llcrypt registered [ 44.588887] Key type .llcrypt registered [ 44.620288] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 51.670502] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 52.241742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.247871] alg: No test for adler32 (adler32-zlib) [ 53.267311] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 53.540838] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 55.152117] Key type lgssc registered [ 55.720867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 62.142278] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 63.045897] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 66.484072] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 68.970931] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 71.416064] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 76.284673] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 80.421855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 80.441514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 80.449809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 81.530438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 81.543052] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 81.580046] Lustre: lustre-MDT0000: new disk, initializing [ 81.605430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 81.613556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 82.764633] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 87.902038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 87.935555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 87.965816] Lustre: 6468:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 87.977360] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 87.979015] Lustre: Skipped 1 previous similar message [ 88.021179] Lustre: lustre-MDT0001: new disk, initializing [ 88.042876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 88.050918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 88.054633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 89.417225] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 91.755376] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 94.962204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 94.992896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 95.079958] Lustre: lustre-OST0000: new disk, initializing [ 95.082281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 95.102494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 97.052555] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 97.195468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 97.198741] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 97.219808] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 102.497353] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 102.528843] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 102.565484] Lustre: lustre-OST0001: new disk, initializing [ 102.567179] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 102.585057] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 104.491593] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 109.036664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 109.039100] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 109.058610] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 110.319530] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 118.047043] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 124.874860] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing check_logdir /tmp/testlogs/ [ 126.317418] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing yml_node [ 127.938706] Lustre: DEBUG MARKER: Client: 2.16.57.62 [ 128.846706] Lustre: DEBUG MARKER: MDS: 2.16.57.62 [ 129.737604] Lustre: DEBUG MARKER: OSS: 2.16.57.62 [ 130.292252] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Aug 19 16:56:35 EDT 2025 [ 135.842882] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 136.310258] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 149.472625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 149.474928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 149.478794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 149.985041] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 149.985585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 149.989946] Lustre: Skipped 1 previous similar message [ 149.993696] Lustre: Skipped 2 previous similar messages [ 155.104926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 155.107413] Lustre: Skipped 3 previous similar messages [ 160.224635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 160.227177] Lustre: Skipped 3 previous similar messages [ 162.272109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 162.311706] LustreError: 12128:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 162.340061] Lustre: server umount lustre-MDT0000 complete [ 163.469763] LustreError: 8361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637028 with bad export cookie 1029602421810579732 [ 163.471105] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 163.472870] LustreError: 8361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 165.344778] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 165.345061] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 165.349698] Lustre: Skipped 2 previous similar messages [ 169.597033] LustreError: 12334:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 169.598938] LustreError: 12334:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 169.671809] Lustre: server umount lustre-MDT0001 complete [ 171.001079] LustreError: 12535:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 171.004021] LustreError: 12535:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 171.024541] Lustre: server umount lustre-OST0000 complete [ 172.386704] Lustre: server umount lustre-OST0001 complete [ 174.483266] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 177.485726] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 181.264737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 183.544317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 185.683546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 187.910734] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 190.324468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 190.345863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 190.430525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 190.445073] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 190.480852] Lustre: lustre-MDT0000: new disk, initializing [ 190.505680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 190.512404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 191.837132] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 196.080944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 196.105589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 196.135335] Lustre: 14901:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 196.240635] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 197.508854] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 200.655198] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 202.042908] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 204.355917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 204.374759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 204.467422] Lustre: lustre-OST0000: new disk, initializing [ 204.468917] Lustre: Skipped 1 previous similar message [ 204.470714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 204.472748] Lustre: Skipped 2 previous similar messages [ 204.501186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.505143] Lustre: Skipped 1 previous similar message [ 205.561896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 205.564658] Lustre: Skipped 1 previous similar message [ 205.567052] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 205.604621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 206.404959] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 209.537754] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 210.923861] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 211.000468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 212.358285] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 212.437682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 215.150187] LustreError: 17813:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 215.153147] LustreError: 17813:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 215.216148] Lustre: server umount lustre-OST0000 complete [ 217.056976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 217.059171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 217.063403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 217.065140] Lustre: Skipped 3 previous similar messages [ 221.154218] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 226.277084] LustreError: 18013:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 226.278958] LustreError: 18013:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 226.302946] Lustre: server umount lustre-MDT0000 complete [ 227.578533] LustreError: 16637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637092 with bad export cookie 1029602421810581251 [ 227.581957] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 227.582160] LustreError: 16637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 238.052190] LustreError: 18215:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 238.054220] LustreError: 18215:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 238.137331] Lustre: server umount lustre-MDT0001 complete [ 239.537843] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 16:58:24 (1755637104) [ 240.366036] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 240.963601] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 16:58:25 (1755637105) [ 243.214897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 243.365405] LustreError: 18885:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 243.374516] LustreError: 18885:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 243.389728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 244.628292] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 247.467017] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 248.944810] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 250.820652] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 252.102290] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 254.482784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 256.600933] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 258.429663] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 273.378179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 273.380643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 273.385218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 273.387767] Lustre: Skipped 4 previous similar messages [ 277.876065] LustreError: 21132:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 277.877708] LustreError: 21132:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 277.947378] Lustre: server umount lustre-OST0000 complete [ 285.329898] Lustre: server umount lustre-MDT0000 complete [ 286.518343] LustreError: 19596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637151 with bad export cookie 1029602421810582133 [ 286.521757] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 286.522148] LustreError: 19596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 286.669754] Lustre: server umount lustre-MDT0001 complete [ 290.433663] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 291.516869] Key type lgssc unregistered [ 291.651314] LNet: 22063:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 291.654071] LNetError: 22063:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 291.663325] LNet: Removed LNI 192.168.204.132@tcp [ 291.992148] Key type .llcrypt unregistered [ 291.993214] Key type ._llcrypt unregistered [ 300.556502] Key type ._llcrypt registered [ 300.557716] Key type .llcrypt registered [ 300.596440] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 300.937778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 300.946983] alg: No test for adler32 (adler32-zlib) [ 301.799673] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 301.888439] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 303.480180] Key type lgssc registered [ 303.831432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 306.500263] loop: module loaded [ 306.524600] loop0: detected capacity change from 0 to 204800000 [ 306.586288] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 306.590232] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 306.722561] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 306.728410] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 306.751039] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 306.770573] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 306.784848] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 306.812239] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 308.759505] loop0: detected capacity change from 0 to 204800000 [ 308.815221] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 308.819603] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 308.825324] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 308.873163] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 310.497317] loop0: detected capacity change from 0 to 204800000 [ 312.413730] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 312.432358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 312.437132] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 313.526324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 313.540406] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 313.578570] Lustre: lustre-MDT0000: new disk, initializing [ 313.603070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 313.611699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 314.973603] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 318.619268] loop1: detected capacity change from 0 to 204800000 [ 320.552964] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 320.574642] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 320.676186] Lustre: lustre-OST07c6: new disk, initializing [ 320.678442] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 320.680619] Lustre: Skipped 1 previous similar message [ 320.702706] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 322.609485] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 322.629476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 322.632252] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 322.642354] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 403.245805] LustreError: 26409:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 403.313366] Lustre: server umount lustre-OST07c6 complete [ 416.740038] LustreError: 26909:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 416.742521] LustreError: 26909:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 416.839208] Lustre: server umount lustre-MDT0000 complete [ 422.634455] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 423.799531] Key type lgssc unregistered [ 423.950422] LNet: 27738:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 423.952801] LNetError: 27738:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 423.961282] LNet: Removed LNI 192.168.204.132@tcp [ 424.286831] Key type .llcrypt unregistered [ 424.288734] Key type ._llcrypt unregistered [ 431.818668] Key type ._llcrypt registered [ 431.819705] Key type .llcrypt registered [ 431.860463] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 432.456961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 432.470389] alg: No test for adler32 (adler32-zlib) [ 433.345361] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 433.438822] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 435.032228] Key type lgssc registered [ 435.496530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 437.984361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 437.990565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 439.126686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 440.251655] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 442.964280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 443.083977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 444.365185] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 446.194314] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 447.442828] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 449.691160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 449.806403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 451.629860] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 453.320026] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 464.167586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 470.665511] LustreError: 31162:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 473.569137] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 473.571045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 473.576132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 474.594246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 474.594283] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 474.594538] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 474.596290] Lustre: Skipped 1 previous similar message [ 479.407119] LustreError: 31414:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 479.470191] Lustre: server umount lustre-OST0000 complete [ 484.320867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 484.323194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 484.327266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 486.972920] LustreError: 31616:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 486.974759] LustreError: 31616:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 487.005890] Lustre: server umount lustre-MDT0000 complete [ 488.275272] LustreError: 28941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637353 with bad export cookie 1289631541611564686 [ 488.281048] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 488.362309] LustreError: 31818:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 488.364051] LustreError: 31818:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 488.457773] Lustre: server umount lustre-MDT0001 complete [ 492.084583] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 493.167718] Key type lgssc unregistered [ 493.306391] LNet: 32344:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 493.309881] LNetError: 32344:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 493.319291] LNet: Removed LNI 192.168.204.132@tcp [ 493.619192] Key type .llcrypt unregistered [ 493.621059] Key type ._llcrypt unregistered [ 501.065425] Key type ._llcrypt registered [ 501.066652] Key type .llcrypt registered [ 501.105433] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 501.482508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 501.553430] alg: No test for adler32 (adler32-zlib) [ 502.409625] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 502.494223] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 504.080173] Key type lgssc registered [ 504.512894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 507.250330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 507.257816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 508.420827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 509.740392] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 512.583290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 512.704050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 513.976354] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 515.711350] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 516.915101] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 519.167165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 519.275376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 521.065883] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 522.842558] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 534.631436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 543.201195] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 543.203684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 543.207818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 545.251389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 545.252523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 545.258524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 548.141190] LustreError: 35812:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 548.219986] Lustre: server umount lustre-OST0000 complete [ 550.369914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 550.376585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 550.378665] Lustre: Skipped 2 previous similar messages [ 555.489254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 555.492753] Lustre: Skipped 1 previous similar message [ 555.579616] LustreError: 36017:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 555.581744] LustreError: 36017:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 555.613934] Lustre: server umount lustre-MDT0000 complete [ 556.896194] LustreError: 35814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637421 with bad export cookie 7241575171555497620 [ 556.899510] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 556.900889] LustreError: 35814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 556.970640] LustreError: 36220:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 556.973253] LustreError: 36220:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 557.061592] Lustre: server umount lustre-MDT0001 complete [ 560.957330] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 562.032058] Key type lgssc unregistered [ 562.168391] LNet: 36746:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 562.170770] LNetError: 36746:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 562.184336] LNet: Removed LNI 192.168.204.132@tcp [ 562.498890] Key type .llcrypt unregistered [ 562.500047] Key type ._llcrypt unregistered [ 571.164487] Key type ._llcrypt registered [ 571.165981] Key type .llcrypt registered [ 571.204375] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 571.495250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 571.504883] alg: No test for adler32 (adler32-zlib) [ 572.356715] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 572.440529] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 574.024115] Key type lgssc registered [ 574.371369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 576.895451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 576.902038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 578.058523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 579.347787] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 582.157649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 582.278251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 583.590675] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 585.451289] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 586.652874] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 589.005209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 589.122846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 591.073668] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 592.924245] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 602.471958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 607.715458] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 607.716723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 607.718831] Lustre: Skipped 2 previous similar messages [ 607.721205] Lustre: Skipped 1 previous similar message [ 607.808955] LustreError: 40361:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 607.840521] Lustre: server umount lustre-MDT0000 complete [ 608.510160] LustreError: 38109:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 608.515682] LustreError: 38109:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 609.076093] LustreError: 38094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637473 with bad export cookie 6031787914627731306 [ 609.079231] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 609.080842] LustreError: 38094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 611.206572] LustreError: 40562:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 611.209825] LustreError: 40562:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 611.282216] Lustre: server umount lustre-MDT0001 complete [ 622.205739] LustreError: 38094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.32@tcp arrived at 1755637487 with bad export cookie 6031787914627731957 [ 622.209834] LustreError: 38094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 638.380501] LustreError: 40784:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 638.382484] LustreError: 40784:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 638.463759] Lustre: server umount lustre-OST0000 complete [ 643.995164] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 645.108843] Key type lgssc unregistered [ 645.249415] LNet: 41616:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 645.253378] LNetError: 41616:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 645.259374] LNet: Removed LNI 192.168.204.132@tcp [ 645.624953] Key type .llcrypt unregistered [ 645.626382] Key type ._llcrypt unregistered [ 654.500989] Key type ._llcrypt registered [ 654.502832] Key type .llcrypt registered [ 654.543381] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 654.853539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 654.876568] alg: No test for adler32 (adler32-zlib) [ 655.730030] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 655.809910] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 657.392097] Key type lgssc registered [ 657.743983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 660.353654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 660.361346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 661.524624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 662.860499] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 665.646432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 665.784854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 667.144566] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 668.864489] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 670.032486] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 672.212260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 672.344514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 674.247868] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 675.999886] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 687.719923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 696.289257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 696.291429] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 696.295521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 698.339352] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 698.339669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 698.340934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 698.348367] Lustre: Skipped 2 previous similar messages [ 699.759104] LustreError: 45237:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 699.827670] Lustre: server umount lustre-OST0000 complete [ 712.416698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 712.420312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 717.281169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 717.283443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 717.286968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 718.590495] LustreError: 45612:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 718.592697] LustreError: 45612:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 718.625384] Lustre: server umount lustre-MDT0000 complete [ 719.932648] LustreError: 42965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637584 with bad export cookie 1407625437378802608 [ 719.937739] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 719.938441] LustreError: 42965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 726.082358] LustreError: 45814:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 726.083989] LustreError: 45814:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 726.176487] Lustre: server umount lustre-MDT0001 complete [ 729.935372] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 731.017182] Key type lgssc unregistered [ 731.155389] LNet: 46342:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 731.158447] LNetError: 46342:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 731.170443] LNet: Removed LNI 192.168.204.132@tcp [ 731.487439] Key type .llcrypt unregistered [ 731.488514] Key type ._llcrypt unregistered [ 740.628382] Key type ._llcrypt registered [ 740.629810] Key type .llcrypt registered [ 740.669322] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 740.978710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 740.986108] alg: No test for adler32 (adler32-zlib) [ 741.839905] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 741.927412] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 743.512091] Key type lgssc registered [ 743.861753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 746.496889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 746.504420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 747.665380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 748.988326] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 751.915768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 752.056746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 753.437230] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 755.203642] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 756.421195] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 758.686165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 758.809845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 760.588207] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 762.390527] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 772.132074] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 774.910372] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 776.008442] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 786.783627] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-08-1917h07m31s [ 787.332742] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 788.220866] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.32@tcp (stopping) [ 789.238906] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.32@tcp (stopping) [ 792.549562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 792.551207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 792.553146] Lustre: Skipped 2 previous similar messages [ 792.554723] Lustre: Skipped 1 previous similar message [ 794.240368] LustreError: 50275:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 794.278466] Lustre: server umount lustre-MDT0000 complete [ 795.381923] LustreError: 49042:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 795.388575] LustreError: 49042:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 795.459693] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 797.662594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 797.703764] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 797.788445] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 797.811169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 798.822860] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 799.021159] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 800.183843] Lustre: DEBUG MARKER: Wait for df (35961) ... [ 800.735869] Lustre: DEBUG MARKER: done [ 802.785602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 802.785782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 802.787514] LustreError: Skipped 1 previous similar message [ 802.791570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 802.793745] Lustre: Skipped 5 previous similar messages [ 807.788073] LustreError: 51317:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 807.791402] LustreError: 51317:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 807.862408] Lustre: server umount lustre-OST0000 complete [ 809.441733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 809.441773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 809.442441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 809.442447] Lustre: Skipped 2 previous similar messages [ 809.445330] Lustre: Skipped 2 previous similar messages [ 815.235363] LustreError: 51523:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 815.237011] LustreError: 51523:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 815.264335] Lustre: server umount lustre-MDT0000 complete [ 815.521426] LustreError: 47709:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 815.528096] LustreError: 47709:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 816.524487] LustreError: 50006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637681 with bad export cookie 14596408961593635694 [ 816.528202] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 816.685184] Lustre: server umount lustre-MDT0001 complete [ 820.407822] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 821.446712] Key type lgssc unregistered [ 821.581420] LNet: 52249:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.585463] LNetError: 52249:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 821.595512] LNet: Removed LNI 192.168.204.132@tcp [ 821.916538] Key type .llcrypt unregistered [ 821.917820] Key type ._llcrypt unregistered [ 833.002167] Key type ._llcrypt registered [ 833.003521] Key type .llcrypt registered [ 833.044078] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 839.487030] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 839.913711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 839.935562] alg: No test for adler32 (adler32-zlib) [ 840.793545] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 840.887397] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 842.472222] Key type lgssc registered [ 842.853215] Lustre: Echo OBD driver; http://www.lustre.org/ [ 846.076858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 848.409995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 850.491847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 852.620177] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 854.921977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 854.948690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 854.953913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 856.038749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 856.051524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 856.090764] Lustre: lustre-MDT0000: new disk, initializing [ 856.115318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 856.122359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 857.454953] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 861.618331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 861.650848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 861.684590] Lustre: 56156:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 861.716388] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 861.719149] Lustre: Skipped 1 previous similar message [ 861.760462] Lustre: lustre-MDT0001: new disk, initializing [ 861.797788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 861.813031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 861.818081] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 863.143507] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 866.229388] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 867.457634] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 869.740519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 869.763379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 869.868235] Lustre: lustre-OST0000: new disk, initializing [ 869.869992] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 869.897568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 871.742810] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 874.851204] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 875.514571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 875.517676] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 875.555755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 876.230122] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 876.322370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 877.673649] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 877.755894] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 880.610410] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 880.611181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 880.613565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 880.616729] Lustre: Skipped 1 previous similar message [ 884.461980] LustreError: 59066:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 884.521533] Lustre: server umount lustre-OST0000 complete [ 887.264923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 887.268611] LustreError: Skipped 1 previous similar message [ 887.271069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 887.277292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 887.279396] Lustre: Skipped 1 previous similar message [ 890.849863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 890.850428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 895.969678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 895.971659] Lustre: Skipped 2 previous similar messages [ 900.064107] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 900.102387] LustreError: 59267:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 900.105955] LustreError: 59267:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 900.136711] Lustre: server umount lustre-MDT0000 complete [ 901.373584] LustreError: 56149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637766 with bad export cookie 14110592642172629770 [ 901.376510] LustreError: 56149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 901.376588] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 901.433531] LustreError: 59472:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 901.435151] LustreError: 59472:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 901.518395] Lustre: server umount lustre-MDT0001 complete [ 904.308075] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 17:09:29 (1755637769) [ 906.558737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 906.707259] LustreError: 60141:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 906.713502] LustreError: 60141:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 906.729974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 907.983888] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 910.801759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 910.915411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 912.246769] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 914.045651] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 915.260810] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 917.409396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 919.388061] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 921.175250] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 939.037535] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-08-1917h10m03s [ 939.600995] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 941.874315] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 942.688154] Lustre: DEBUG MARKER: Creating a test file and stat it [ 942.705715] LustreError: 61455:0:(mdt_handler.c:2329:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 962.818427] Lustre: lustre-MDT0000: Client 76d75f65-3445-4014-9058-46c58131834d (at 192.168.204.32@tcp) reconnecting [ 982.752116] LustreError: 61455:0:(mdt_handler.c:2329:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 983.443535] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 984.235453] Lustre: DEBUG MARKER: done [ 987.104826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 987.106876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 987.110369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 991.854655] LustreError: 63146:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 991.856595] LustreError: 63146:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 991.914181] Lustre: server umount lustre-OST0000 complete [ 993.249306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 993.254059] Lustre: Skipped 1 previous similar message [ 999.296769] LustreError: 63348:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 999.300109] LustreError: 63348:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 999.323838] Lustre: server umount lustre-MDT0000 complete [ 999.330591] LustreError: 60159:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 999.336031] LustreError: 60159:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1000.530327] LustreError: 60122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637865 with bad export cookie 14110592642172630624 [ 1000.533676] LustreError: 60122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1000.535438] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1000.691772] Lustre: server umount lustre-MDT0001 complete [ 1004.272889] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 1005.326056] Key type lgssc unregistered [ 1005.446360] LNet: 64079:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.448726] LNetError: 64079:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1005.459358] LNet: Removed LNI 192.168.204.132@tcp [ 1005.764841] Key type .llcrypt unregistered [ 1005.766797] Key type ._llcrypt unregistered [ 1016.662707] Key type ._llcrypt registered [ 1016.663735] Key type .llcrypt registered [ 1016.701553] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 1022.948531] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1023.301801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1023.343234] alg: No test for adler32 (adler32-zlib) [ 1024.205398] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 1024.292542] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 1025.872201] Key type lgssc registered [ 1026.256287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1029.740832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1032.080883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1034.102211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1036.180847] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1038.435550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1038.453056] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1038.458229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1039.540334] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1039.553114] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1039.591569] Lustre: lustre-MDT0000: new disk, initializing [ 1039.614601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1039.622416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1040.889747] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1045.055940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1045.083137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1045.115625] Lustre: 67989:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1045.141297] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1045.142909] Lustre: Skipped 1 previous similar message [ 1045.182507] Lustre: lustre-MDT0001: new disk, initializing [ 1045.221557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1045.238285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1045.244103] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1046.509086] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1049.619229] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1050.803701] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1052.979779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1053.004327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1053.100263] Lustre: lustre-OST0000: new disk, initializing [ 1053.102072] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1053.132528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1055.051351] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1058.098592] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1059.510924] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1062.394600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1062.397577] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1062.429151] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1062.662946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 1063.923913] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1064.004401] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1067.491703] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1067.491982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1067.493689] LustreError: Skipped 1 previous similar message [ 1067.498299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1067.500394] Lustre: Skipped 1 previous similar message [ 1070.700134] LustreError: 70919:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1070.764720] Lustre: server umount lustre-OST0000 complete [ 1072.610638] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1072.611984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1072.618733] Lustre: Skipped 2 previous similar messages [ 1077.730399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1077.733037] Lustre: Skipped 2 previous similar messages [ 1078.144290] LustreError: 71121:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1078.146701] LustreError: 71121:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1078.177231] Lustre: server umount lustre-MDT0000 complete [ 1079.434572] LustreError: 68764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637944 with bad export cookie 5140097657871054187 [ 1079.439923] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1079.506410] LustreError: 71323:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1079.508400] LustreError: 71323:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1079.606284] Lustre: server umount lustre-MDT0001 complete [ 1082.331779] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 17:12:27 (1755637947) [ 1082.848112] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1083.420554] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 17:12:28 (1755637948) [ 1087.341208] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1088.760841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1088.937561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1089.292777] Lustre: Failing over lustre-MDT0000 [ 1089.295105] LustreError: 72392:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1089.299228] LustreError: 72392:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1089.301774] LustreError: 72392:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 1089.371073] LustreError: 72443:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1089.372977] LustreError: 72443:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1089.459972] Lustre: server umount lustre-MDT0000 complete [ 1091.584319] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 17:12:36 (1755637956) [ 1093.797612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1093.935731] LustreError: 73009:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1093.942887] LustreError: 73009:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1093.950161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1095.264911] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1098.073468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1099.492438] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1101.375771] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1102.593459] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1104.819083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1104.924700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1104.926830] Lustre: Skipped 1 previous similar message [ 1106.834361] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1108.656588] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1121.006509] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1123.298983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1123.300051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1123.302307] Lustre: Skipped 2 previous similar messages [ 1123.305310] Lustre: Skipped 1 previous similar message [ 1128.417881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1128.420383] Lustre: Skipped 2 previous similar messages [ 1128.766865] LustreError: 75355:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1128.769799] LustreError: 75355:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1128.801235] Lustre: server umount lustre-MDT0000 complete [ 1130.151559] LustreError: 72990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755637995 with bad export cookie 5140097657871055328 [ 1130.155210] LustreError: 72990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1130.157591] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1130.305774] Lustre: server umount lustre-MDT0001 complete [ 1131.410635] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1132.882197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1135.765793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1140.704845] LustreError: 66202:0:(client.c:1385:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000423a798d x1840919755153792/t0(0) o250->MGC192.168.204.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1140.840825] LustreError: 74603:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1140.887772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1140.927426] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1142.319805] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1145.305196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1145.407986] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1146.908266] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1148.760469] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1149.536131] Lustre: 66205:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1755637998/real 1755637998] req@00000000b6987f42 x1840919755153408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1755638014 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1149.858257] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1150.952351] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1152.480143] Lustre: 66203:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1755638000/real 1755638000] req@000000006c534b25 x1840919755153536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1755638016 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1154.021243] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1155.040185] Lustre: 66204:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1755638003/real 1755638003] req@00000000d5d9a0ef x1840919755153664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1755638019 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1159.139739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1159.141954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1159.146701] Lustre: Skipped 2 previous similar messages [ 1159.148904] Lustre: Skipped 1 previous similar message [ 1163.075162] LustreError: 77803:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1163.078296] LustreError: 77803:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1163.109801] Lustre: server umount lustre-MDT0000 complete [ 1164.258209] LustreError: 76414:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.264161] LustreError: 76414:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1164.541475] LustreError: 72989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638029 with bad export cookie 5140097657871057218 [ 1164.546645] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1164.546736] LustreError: 72989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1164.720218] Lustre: server umount lustre-MDT0001 complete [ 1166.985763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1170.395766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1174.497063] LustreError: 66202:0:(client.c:1385:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000007482814 x1840919755174144/t0(0) o250->MGC192.168.204.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1174.637659] LustreError: 76436:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1174.666251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1174.668640] Lustre: Skipped 1 previous similar message [ 1174.688500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1176.005070] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1179.011983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1179.108636] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1180.434641] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1182.250081] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1183.497520] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1184.235740] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1185.248262] Lustre: 66206:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1755638034/real 1755638034] req@00000000666848a0 x1840919755173888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1755638050 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1185.259973] Lustre: 66206:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1189.358583] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1190.368097] Lustre: 66206:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1755638039/real 1755638039] req@00000000edba311a x1840919755174016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1755638055 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1194.467735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1194.467977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1194.472889] Lustre: Skipped 3 previous similar messages [ 1196.931243] LustreError: 80247:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1196.933207] LustreError: 80247:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1196.961881] Lustre: server umount lustre-MDT0000 complete [ 1198.249399] LustreError: 72991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638063 with bad export cookie 5140097657871058590 [ 1198.255196] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1198.395329] Lustre: server umount lustre-MDT0001 complete [ 1200.560262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1201.658525] Lustre: DEBUG MARKER: files compared the same [ 1212.666262] Lustre: server umount lustre-OST0000 complete [ 1218.233882] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 1219.286971] Key type lgssc unregistered [ 1219.420389] LNet: 81872:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1219.422911] LNetError: 81872:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1219.432306] LNet: Removed LNI 192.168.204.132@tcp [ 1219.740700] Key type .llcrypt unregistered [ 1219.741957] Key type ._llcrypt unregistered [ 1228.723633] Key type ._llcrypt registered [ 1228.725639] Key type .llcrypt registered [ 1228.761418] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1229.072060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1229.096069] alg: No test for adler32 (adler32-zlib) [ 1229.945831] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 1230.032466] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 1231.608115] Key type lgssc registered [ 1231.958444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1234.487446] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1234.494579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1235.646556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1236.957079] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug +malloc all [ 1239.794080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1239.892084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1241.270458] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug +malloc all [ 1243.080810] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1244.299908] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1246.706053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1246.809814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1248.785110] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug +malloc all [ 1250.649356] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1261.157077] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1265.632713] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1265.636037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1265.639463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1266.657279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1266.658050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1266.658205] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1269.801758] LustreError: 85488:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1269.856228] Lustre: server umount lustre-OST0000 complete [ 1271.776693] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1271.783319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1271.784916] Lustre: Skipped 2 previous similar messages [ 1276.896836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1276.900690] Lustre: Skipped 1 previous similar message [ 1277.130728] LustreError: 85690:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1277.134246] LustreError: 85690:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1277.167219] Lustre: server umount lustre-MDT0000 complete [ 1278.418736] LustreError: 84845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638143 with bad export cookie 811807134696632940 [ 1278.420667] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1278.424857] LustreError: 84845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1278.496109] LustreError: 85893:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1278.499263] LustreError: 85893:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1278.570406] Lustre: server umount lustre-MDT0001 complete [ 1282.254657] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 1283.362871] Key type lgssc unregistered [ 1283.484328] LNet: 86420:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1283.487490] LNetError: 86420:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1283.494357] LNet: Removed LNI 192.168.204.132@tcp [ 1283.775685] Key type .llcrypt unregistered [ 1283.776975] Key type ._llcrypt unregistered [ 1292.617626] Key type ._llcrypt registered [ 1292.619050] Key type .llcrypt registered [ 1292.662919] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1293.121120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1293.195888] alg: No test for adler32 (adler32-zlib) [ 1294.059742] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 1294.157213] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 1295.744236] Key type lgssc registered [ 1296.145790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1298.985765] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1298.991477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1316.512317] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1322.656214] LustreError: 87760:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.132@tcp: failed processing log, type 4: rc = -110 [ 1352.352174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1354.250721] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1356.054802] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1358.651516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1358.689372] Lustre: *** cfs_fail_loc=706, val=0*** [ 1358.850063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1358.874058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1360.137458] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1363.109096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1363.240607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1364.682642] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1366.407949] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1367.657872] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1368.545280] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1368.548071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1368.553334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1369.057188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1369.061897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1369.068966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1370.594761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1370.596674] Lustre: Skipped 1 previous similar message [ 1374.446253] LustreError: 90070:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1374.507206] Lustre: server umount lustre-OST0000 complete [ 1378.784864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1378.787975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1378.792668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1380.834057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1385.953070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1385.955818] Lustre: Skipped 3 previous similar messages [ 1390.048140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1390.081536] LustreError: 90271:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1390.083385] LustreError: 90271:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1390.112134] Lustre: server umount lustre-MDT0000 complete [ 1391.073856] LustreError: 88732:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1391.083458] LustreError: 88732:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1391.403506] LustreError: 87767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638256 with bad export cookie 9808273603884270089 [ 1391.406272] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1391.407784] LustreError: 87767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1391.473573] LustreError: 90476:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1391.476323] LustreError: 90476:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1391.553113] Lustre: server umount lustre-MDT0001 complete [ 1395.310736] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 1396.534920] Key type lgssc unregistered [ 1396.676524] LNet: 91002:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1396.679263] LNetError: 91002:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1396.689337] LNet: Removed LNI 192.168.204.132@tcp [ 1397.009812] Key type .llcrypt unregistered [ 1397.011166] Key type ._llcrypt unregistered [ 1406.236352] Key type ._llcrypt registered [ 1406.237848] Key type .llcrypt registered [ 1406.278353] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1406.581634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1406.587626] alg: No test for adler32 (adler32-zlib) [ 1407.431963] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 1407.512702] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 1409.088141] Key type lgssc registered [ 1409.433603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1411.981267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1411.988052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1414.423558] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1416.935267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1417.113443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1418.451409] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1421.291371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1421.398132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1423.258934] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1426.081904] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1426.086063] Lustre: MGS: Client efa5930f-4308-48e9-8cb4-d6539ad8a5a1 (at 0@lo) reconnecting [ 1426.165610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1426.200642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1427.442215] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1436.641132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1436.643293] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1436.647078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1437.667706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1437.668172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1437.675731] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1441.326083] LustreError: 94978:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1441.381116] Lustre: server umount lustre-OST0000 complete [ 1442.785581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1442.785609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1442.793817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1442.795560] Lustre: Skipped 2 previous similar messages [ 1447.905266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1447.907226] Lustre: Skipped 1 previous similar message [ 1448.702089] LustreError: 95180:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1448.704417] LustreError: 95180:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1448.714899] Lustre: server umount lustre-MDT0000 complete [ 1449.999266] LustreError: 95383:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1450.001845] LustreError: 95383:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1450.088912] Lustre: server umount lustre-MDT0001 complete [ 1451.343213] Lustre: server umount MGS complete [ 1454.951353] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 17:18:39 (1755638319) [ 1464.371458] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 1467.419088] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1470.573938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1472.746787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1474.790384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1476.924819] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1479.135867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1479.158522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1480.411338] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1484.268780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1484.291811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1484.366615] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1484.384066] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1484.420777] Lustre: lustre-MDT0001: new disk, initializing [ 1484.445362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1484.454059] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1484.456440] Lustre: 100074:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1484.459682] LustreError: 100074:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1485.741513] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1489.858037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1489.880661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1489.968110] Lustre: lustre-OST0000: new disk, initializing [ 1489.970204] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1489.991201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1491.820665] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1494.787692] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1497.040976] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1497.045094] Lustre: MGS: Client 29571b0e-71a2-4ebf-b879-0250995fe5fa (at 0@lo) reconnecting [ 1497.058742] Lustre: 99364:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1497.065825] LustreError: 99364:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1497.076940] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1497.112079] Lustre: lustre-MDT0000: new disk, initializing [ 1497.161536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1498.209262] Lustre: 101771:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@00000000c851a3eb x1840920156736640/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1755638379 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1498.214237] Lustre: 101771:0:(client.c:1611:after_reply()) Skipped 1 previous similar message [ 1498.383517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1498.388048] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1498.403500] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1500.259372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 1500.259398] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1500.261786] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1517.538126] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1517.542451] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1517.547518] Lustre: Skipped 1 previous similar message [ 1517.550440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1529.313988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1529.315830] Lustre: Skipped 5 previous similar messages [ 1531.360141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1531.381073] LustreError: 102287:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1531.382793] LustreError: 102287:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1531.461454] Lustre: server umount lustre-OST0000 complete [ 1534.433963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1534.434136] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1534.438484] Lustre: Skipped 1 previous similar message [ 1534.441369] LustreError: Skipped 1 previous similar message [ 1547.232117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1547.263086] LustreError: 102491:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 1547.265298] LustreError: 102491:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1547.276518] Lustre: server umount lustre-MDT0000 complete [ 1554.698850] Lustre: server umount lustre-MDT0001 complete [ 1555.923036] LustreError: 102894:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1555.924990] LustreError: 102894:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1555.949737] Lustre: server umount MGS complete [ 1559.544179] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 17:20:24 (1755638424) [ 1565.340130] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 1566.427520] Key type lgssc unregistered [ 1566.567315] LNet: 104222:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1566.569769] LNetError: 104222:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1566.576366] LNet: Removed LNI 192.168.204.132@tcp [ 1566.864404] Key type .llcrypt unregistered [ 1566.866346] Key type ._llcrypt unregistered [ 1573.795871] Key type ._llcrypt registered [ 1573.797303] Key type .llcrypt registered [ 1573.835964] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1574.158756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1574.190059] alg: No test for adler32 (adler32-zlib) [ 1575.053140] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 1575.148916] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 1576.728126] Key type lgssc registered [ 1577.080494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1579.623173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1579.631164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1580.781046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1582.049360] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1584.812125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1584.929952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1586.275927] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1588.049625] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1589.232723] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1591.415419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1591.528811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1593.407156] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1595.214508] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1606.888530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1615.329120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1615.331373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1615.334919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1617.378323] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1617.379084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1617.380225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1617.385807] Lustre: Skipped 1 previous similar message [ 1619.630117] LustreError: 107645:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1619.690270] Lustre: server umount lustre-OST0000 complete [ 1622.498314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1622.498933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1622.505938] Lustre: Skipped 1 previous similar message [ 1627.617172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1627.619847] Lustre: Skipped 2 previous similar messages [ 1632.737896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1632.740516] Lustre: Skipped 1 previous similar message [ 1635.296138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1635.327846] LustreError: 107847:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1635.329485] LustreError: 107847:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1635.362328] Lustre: server umount lustre-MDT0000 complete [ 1636.608958] LustreError: 105375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638501 with bad export cookie 2415200361545007572 [ 1636.611964] LustreError: 105375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1636.612440] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1636.674614] LustreError: 108051:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1636.676224] LustreError: 108051:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1636.759434] Lustre: server umount lustre-MDT0001 complete [ 1640.306513] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 1641.403596] Key type lgssc unregistered [ 1641.546388] LNet: 108577:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1641.549218] LNetError: 108577:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1641.557295] LNet: Removed LNI 192.168.204.132@tcp [ 1641.881864] Key type .llcrypt unregistered [ 1641.884528] Key type ._llcrypt unregistered [ 1648.238385] Key type ._llcrypt registered [ 1648.239726] Key type .llcrypt registered [ 1648.281598] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1648.691237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1648.763179] alg: No test for adler32 (adler32-zlib) [ 1649.623433] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 1649.700829] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 1651.272127] Key type lgssc registered [ 1651.597343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1653.678953] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1653.681924] LustreError: 109611:0:(tgt_mount.c:2375:server_fill_super()) cfs_race id 716 sleeping [ 1653.683199] Lustre: Skipped 1 previous similar message [ 1653.688131] LustreError: 109610:0:(tgt_mount.c:2375:server_fill_super()) cfs_fail_race id 716 waking [ 1653.691943] LustreError: 109611:0:(tgt_mount.c:2375:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 1653.694071] LustreError: 109611:0:(tgt_mount.c:2387:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1653.697938] LustreError: 109611:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1653.698502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1654.853402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1657.425619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1657.519521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1658.844708] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1660.389615] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing lsmod [ 1661.406054] LustreError: 110775:0:(tgt_mount.c:2375:server_fill_super()) cfs_race id 716 sleeping [ 1661.418985] LustreError: 110778:0:(tgt_mount.c:2375:server_fill_super()) cfs_fail_race id 716 waking [ 1661.421092] LustreError: 110775:0:(tgt_mount.c:2375:server_fill_super()) cfs_fail_race id 716 awake: rc=4987 [ 1661.423259] LustreError: 110775:0:(tgt_mount.c:2387:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1661.426363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1661.426850] LustreError: 110775:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1661.546831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1662.945062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1662.947522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1662.952381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1667.043363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1667.044618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1667.046938] Lustre: Skipped 1 previous similar message [ 1667.053767] Lustre: Skipped 1 previous similar message [ 1668.067088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1668.071481] Lustre: Skipped 2 previous similar messages [ 1668.623467] LustreError: 110967:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1668.656516] Lustre: server umount lustre-MDT0000 complete [ 1670.139996] LustreError: 109626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638535 with bad export cookie 13484886938551220317 [ 1670.146298] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1670.147319] LustreError: 109626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1672.160288] LustreError: 110085:0:(client.c:1375:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000063f1e3fc x1840920410471424/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1672.173620] LustreError: 111172:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1672.175889] LustreError: 111172:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1672.262872] Lustre: server umount lustre-MDT0001 complete [ 1683.762853] LustreError: 111372:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1683.765410] LustreError: 111372:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1683.824441] Lustre: server umount lustre-OST0000 complete [ 1686.606664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1686.763286] LustreError: 111849:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1686.769021] LustreError: 111849:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1686.790462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1688.059252] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1690.967621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1691.104950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1692.429601] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1694.359275] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1695.641955] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1697.937125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1699.985551] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1701.786810] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1711.408467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1715.681319] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1715.684596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1715.690829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1716.710676] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1719.917086] LustreError: 114097:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1719.918802] LustreError: 114097:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1720.007378] Lustre: server umount lustre-OST0000 complete [ 1721.825551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1721.825797] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1721.826291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1721.826296] Lustre: Skipped 2 previous similar messages [ 1721.834749] Lustre: Skipped 2 previous similar messages [ 1727.419887] LustreError: 114299:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1727.421787] LustreError: 114299:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1727.447150] Lustre: server umount lustre-MDT0000 complete [ 1728.688581] LustreError: 111830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638593 with bad export cookie 13484886938551220975 [ 1728.692410] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1728.693756] LustreError: 111830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1728.833479] Lustre: server umount lustre-MDT0001 complete [ 1732.600291] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 1733.749132] Key type lgssc unregistered [ 1733.881436] LNet: 115028:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1733.884722] LNetError: 115028:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1733.897479] LNet: Removed LNI 192.168.204.132@tcp [ 1734.294689] Key type .llcrypt unregistered [ 1734.295970] Key type ._llcrypt unregistered [ 1744.909225] Key type ._llcrypt registered [ 1744.910502] Key type .llcrypt registered [ 1744.950424] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1745.238194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1745.245879] alg: No test for adler32 (adler32-zlib) [ 1746.097217] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 1746.179178] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 1747.760190] Key type lgssc registered [ 1748.170824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1750.803163] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1750.809313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1751.984199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1753.568289] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1756.867044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1757.019756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1758.374138] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1760.291750] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1761.712746] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1764.251789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1764.383171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1766.540150] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1768.477673] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1776.679879] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1780.915382] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1781.740477] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1786.853923] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1786.854672] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1786.858916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1786.860951] LustreError: Skipped 1 previous similar message [ 1786.864798] Lustre: Skipped 1 previous similar message [ 1788.912608] LustreError: 118742:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1788.975743] Lustre: server umount lustre-OST0000 complete [ 1791.969950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1791.970565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1791.973441] Lustre: Skipped 2 previous similar messages [ 1791.976762] Lustre: Skipped 1 previous similar message [ 1796.347705] LustreError: 118947:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1796.351456] LustreError: 118947:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1796.375530] Lustre: server umount lustre-MDT0000 complete [ 1797.089410] LustreError: 117593:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1797.094161] LustreError: 117593:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1797.617856] LustreError: 116377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638662 with bad export cookie 9352093389227978183 [ 1797.621347] LustreError: 116377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1797.621855] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1797.693837] LustreError: 119150:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1797.695584] LustreError: 119150:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1797.790443] Lustre: server umount lustre-MDT0001 complete [ 1801.563228] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 1802.771849] Key type lgssc unregistered [ 1802.923508] LNet: 119675:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1802.926679] LNetError: 119675:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1802.935648] LNet: Removed LNI 192.168.204.132@tcp [ 1803.277209] Key type .llcrypt unregistered [ 1803.278762] Key type ._llcrypt unregistered [ 1810.880738] Key type ._llcrypt registered [ 1810.881809] Key type .llcrypt registered [ 1810.923625] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1811.298178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1811.305427] alg: No test for adler32 (adler32-zlib) [ 1812.178808] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 1812.265310] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 1813.848205] Key type lgssc registered [ 1814.252468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1817.086982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1817.094652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1818.261785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1819.640953] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1822.637066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1822.790346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1824.133090] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1826.293496] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1827.774185] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1830.834906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1831.000509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1833.109341] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1835.206970] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1843.302934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1848.802596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1848.802963] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1848.804063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1848.807371] Lustre: Skipped 1 previous similar message [ 1850.224117] LustreError: 123099:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1850.288557] Lustre: server umount lustre-OST0000 complete [ 1853.408973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1853.411574] LustreError: Skipped 1 previous similar message [ 1853.412951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1853.417192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1853.419421] Lustre: Skipped 1 previous similar message [ 1853.627948] LustreError: 123300:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1853.629787] LustreError: 123300:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1853.656003] Lustre: server umount lustre-MDT0000 complete [ 1853.921815] LustreError: 120844:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1853.927690] LustreError: 120844:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1855.061420] LustreError: 122453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638719 with bad export cookie 1577828456071497324 [ 1855.066000] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1855.069171] LustreError: 122453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1855.157689] LustreError: 123502:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1855.160861] LustreError: 123502:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1855.250415] Lustre: server umount lustre-MDT0001 complete [ 1859.761431] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 1861.305247] Key type lgssc unregistered [ 1861.523702] LNet: 124029:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1861.528873] LNetError: 124029:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1861.537644] LNet: Removed LNI 192.168.204.132@tcp [ 1862.076112] Key type .llcrypt unregistered [ 1862.078709] Key type ._llcrypt unregistered [ 1871.663479] Key type ._llcrypt registered [ 1871.664703] Key type .llcrypt registered [ 1871.697263] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1872.006294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1872.045849] alg: No test for adler32 (adler32-zlib) [ 1872.964833] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 1873.059493] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 1874.640215] Key type lgssc registered [ 1875.141657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1878.335615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1878.341598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1879.511494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1881.095774] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1884.214648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1884.342594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1885.863153] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1887.703640] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1888.933909] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1891.435139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1891.545435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1893.492400] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1895.401874] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1904.875275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 1908.836224] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1910.207839] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1910.212187] Lustre: Skipped 2 previous similar messages [ 1912.029721] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1912.032090] Lustre: Skipped 2 previous similar messages [ 1917.981921] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1917.998671] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1922.011716] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.32@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1922.020281] Lustre: Skipped 2 previous similar messages [ 1923.792643] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.32@tcp 192.168.0.[2,10]@tcp [ 1927.474833] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.32/24@tcp in log lustre-MDT0000 [ 1927.481474] Lustre: Skipped 2 previous similar messages [ 1933.819331] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.32/24@tcp [ 1933.821032] Lustre: Skipped 1 previous similar message [ 1940.968175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1940.969198] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1940.971396] LustreError: Skipped 1 previous similar message [ 1940.974413] Lustre: Skipped 1 previous similar message [ 1940.976273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1940.980063] Lustre: Skipped 1 previous similar message [ 1941.932099] LustreError: 129188:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1942.014661] Lustre: server umount lustre-OST0000 complete [ 1946.082986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.083224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1946.083940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1946.088478] Lustre: Skipped 1 previous similar message [ 1946.091975] LustreError: Skipped 2 previous similar messages [ 1949.711778] LustreError: 129390:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1949.715654] LustreError: 129390:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1949.749910] Lustre: server umount lustre-MDT0000 complete [ 1951.202301] LustreError: 126688:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1951.211252] LustreError: 126688:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1951.256087] LustreError: 125376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638816 with bad export cookie 2045021010168865573 [ 1951.263511] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1951.263696] LustreError: 125376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1951.350942] LustreError: 129593:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1951.353377] LustreError: 129593:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1951.447603] Lustre: server umount lustre-MDT0001 complete [ 1955.394166] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 1956.842201] Key type lgssc unregistered [ 1956.998669] LNet: 130119:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1957.003424] LNetError: 130119:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1957.015772] LNet: Removed LNI 192.168.204.132@tcp [ 1957.503045] Key type .llcrypt unregistered [ 1957.504045] Key type ._llcrypt unregistered [ 1968.926211] Key type ._llcrypt registered [ 1968.927324] Key type .llcrypt registered [ 1968.963887] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 1969.283776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1969.343116] alg: No test for adler32 (adler32-zlib) [ 1970.222302] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 1970.323657] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 1971.904243] Key type lgssc registered [ 1972.369595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1975.465000] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1975.479958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1976.661811] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1976.667582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1976.672205] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.32/24@tcp [ 1977.894114] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1982.038413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1982.149351] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1982.158217] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.32/24@tcp [ 1982.175756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1983.627745] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1985.498978] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1986.870702] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1989.410923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1989.535816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1991.577935] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 1993.743978] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2001.834531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 2012.132203] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2012.132448] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2012.133242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2012.138767] Lustre: Skipped 1 previous similar message [ 2014.130164] LustreError: 133939:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2014.240846] Lustre: server umount lustre-OST0000 complete [ 2017.250877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2017.251463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2017.254746] Lustre: Skipped 1 previous similar message [ 2017.257833] Lustre: Skipped 2 previous similar messages [ 2021.952778] LustreError: 134140:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2021.954922] LustreError: 134140:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2021.982509] Lustre: server umount lustre-MDT0000 complete [ 2022.369909] LustreError: 131652:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2022.377804] LustreError: 131652:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2023.283613] LustreError: 132354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638888 with bad export cookie 7913698715338877633 [ 2023.288052] LustreError: 132354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2023.289585] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2023.349635] LustreError: 134342:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2023.351727] LustreError: 134342:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2023.441124] Lustre: server umount lustre-MDT0001 complete [ 2027.388868] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 2028.586697] Key type lgssc unregistered [ 2028.746767] LNet: 134868:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2028.751507] LNetError: 134868:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2028.760570] LNet: Removed LNI 192.168.204.132@tcp [ 2029.119786] Key type .llcrypt unregistered [ 2029.120822] Key type ._llcrypt unregistered [ 2039.727813] Key type ._llcrypt registered [ 2039.728882] Key type .llcrypt registered [ 2039.772172] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 2046.572120] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 2047.040220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2047.083473] alg: No test for adler32 (adler32-zlib) [ 2047.949239] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 2048.043412] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 2049.640241] Key type lgssc registered [ 2050.042466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2053.085615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2055.441973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2057.652152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2059.770474] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2062.352632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2062.374035] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2062.381756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2063.500613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2063.517414] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2063.557564] Lustre: lustre-MDT0000: new disk, initializing [ 2063.586480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2063.595904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2064.919550] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2069.813438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2069.848723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2069.892391] Lustre: 138205:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2069.914751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2069.916496] Lustre: Skipped 1 previous similar message [ 2069.951909] Lustre: lustre-MDT0001: new disk, initializing [ 2069.986490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2070.000758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2070.004739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2071.370911] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2074.422033] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2075.880967] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2078.512249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2078.547809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2078.680677] Lustre: lustre-OST0000: new disk, initializing [ 2078.683133] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2078.722279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2080.974587] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2083.338613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2083.349407] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2083.388084] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2084.258765] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2085.752918] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2085.865161] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2087.287608] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2087.410612] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2088.422670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2088.423136] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2088.427152] LustreError: Skipped 1 previous similar message [ 2088.433763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2088.438243] Lustre: Skipped 1 previous similar message [ 2090.159128] LustreError: 141115:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2090.235481] Lustre: server umount lustre-OST0000 complete [ 2093.541709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2093.542759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2093.548963] Lustre: Skipped 2 previous similar messages [ 2093.555011] Lustre: Skipped 1 previous similar message [ 2097.685877] LustreError: 141316:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2097.688051] LustreError: 141316:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2097.726783] Lustre: server umount lustre-MDT0000 complete [ 2099.130773] LustreError: 138196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755638964 with bad export cookie 4448431033975543531 [ 2099.137746] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2099.231864] LustreError: 141517:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2099.234195] LustreError: 141517:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2099.326799] Lustre: server umount lustre-MDT0001 complete [ 2102.344437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2102.516942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2103.789886] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2106.786515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2106.967690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2108.572545] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2110.383634] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2111.578640] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2113.847687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2115.995093] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2117.799832] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2119.191489] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2127.505348] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2128.907818] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2128.977077] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2132.372511] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2132.402041] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2132.464827] Lustre: lustre-OST0001: new disk, initializing [ 2132.466734] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2132.491751] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2132.494123] Lustre: Skipped 1 previous similar message [ 2134.336789] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2137.516516] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2139.138049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2139.141153] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2139.165125] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2139.656665] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2140.957849] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2141.041366] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2142.286463] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88a887c73800.ost_server_uuid 50 [ 2142.847313] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff88a887c73800.ost_server_uuid in FULL state after 0 sec [ 2149.349294] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2149.349485] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.352153] LustreError: Skipped 1 previous similar message [ 2149.352653] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2149.355494] Lustre: Skipped 1 previous similar message [ 2149.359726] Lustre: Skipped 1 previous similar message [ 2154.469177] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2154.471223] Lustre: Skipped 1 previous similar message [ 2159.072094] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2159.091604] LustreError: 146263:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2159.093438] LustreError: 146263:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2159.111729] Lustre: server umount lustre-OST0001 complete [ 2159.588242] LustreError: 144054:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2159.593162] LustreError: 144054:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 2163.170776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2163.172373] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2163.176368] LustreError: Skipped 1 previous similar message [ 2163.184167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2163.190297] Lustre: Skipped 1 previous similar message [ 2164.712403] LustreError: 146337:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2164.722194] LustreError: 146337:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2166.574432] LustreError: 146467:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2166.576672] LustreError: 146467:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2166.647598] Lustre: server umount lustre-OST0000 complete [ 2168.288834] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2168.291271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2168.295047] Lustre: Skipped 1 previous similar message [ 2174.167082] Lustre: server umount lustre-MDT0000 complete [ 2174.945173] LustreError: 143279:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2174.952381] LustreError: 143279:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2175.372161] LustreError: 141971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639040 with bad export cookie 4448431033975544413 [ 2175.375241] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2175.375828] LustreError: 141971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2175.441709] LustreError: 146870:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2175.443485] LustreError: 146870:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2175.530526] Lustre: server umount lustre-MDT0001 complete [ 2182.262518] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 2183.278390] Key type lgssc unregistered [ 2183.416332] LNet: 147852:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2183.418515] LNetError: 147852:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2183.427324] LNet: Removed LNI 192.168.204.132@tcp [ 2183.714091] Key type .llcrypt unregistered [ 2183.715233] Key type ._llcrypt unregistered [ 2195.942375] Key type ._llcrypt registered [ 2195.945237] Key type .llcrypt registered [ 2195.996692] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 2203.452856] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 2203.860341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2203.943937] alg: No test for adler32 (adler32-zlib) [ 2204.830495] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 2204.931231] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 2206.512217] Key type lgssc registered [ 2207.078646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2211.615174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2214.616978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2217.392700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2220.121348] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2222.473036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2222.489612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2222.495342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2223.575173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2223.586251] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2223.620436] Lustre: lustre-MDT0000: new disk, initializing [ 2223.644294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2223.651274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2225.470884] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2230.578929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2230.626140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2230.677237] Lustre: 151761:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2230.717828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2230.720579] Lustre: Skipped 1 previous similar message [ 2230.758521] Lustre: lustre-MDT0001: new disk, initializing [ 2230.808299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2230.829450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2230.833617] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2232.618566] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2236.053313] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2237.573856] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2240.464683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2240.494301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2240.599493] Lustre: lustre-OST0000: new disk, initializing [ 2240.601961] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2240.648413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2242.841736] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2245.145177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2245.152289] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2245.204413] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2246.492477] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2248.208254] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2248.329680] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2249.918605] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2250.054138] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2255.329985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2255.334303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2255.335960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2255.340674] Lustre: Skipped 1 previous similar message [ 2257.133080] LustreError: 154671:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2257.202135] Lustre: server umount lustre-OST0000 complete [ 2260.452223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2260.453648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2260.457763] Lustre: Skipped 1 previous similar message [ 2260.459941] Lustre: Skipped 1 previous similar message [ 2265.569519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2265.573260] Lustre: Skipped 2 previous similar messages [ 2270.690095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2270.694028] Lustre: Skipped 1 previous similar message [ 2273.248145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2273.280237] LustreError: 154872:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2273.282583] LustreError: 154872:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2273.311731] Lustre: server umount lustre-MDT0000 complete [ 2274.632838] LustreError: 152536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639139 with bad export cookie 17925060048029458332 [ 2274.634601] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2274.635479] LustreError: 152536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2274.710728] LustreError: 155077:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2274.714115] LustreError: 155077:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2274.815568] Lustre: server umount lustre-MDT0001 complete [ 2277.862853] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 17:32:22 (1755639142) [ 2283.395834] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 2286.806219] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 2289.910385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2292.085902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2294.194452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2296.431479] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2298.779157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2298.806338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2298.894133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2298.906719] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2298.951416] Lustre: lustre-MDT0000: new disk, initializing [ 2298.977350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2298.983479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2300.327896] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2304.355650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2304.378555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2304.413849] Lustre: 158049:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2304.434128] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2304.435724] Lustre: Skipped 1 previous similar message [ 2304.469676] Lustre: lustre-MDT0001: new disk, initializing [ 2304.499489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2304.510325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2304.515640] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2305.651126] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2308.813901] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2309.898156] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2312.062313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2312.086177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2313.992224] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2315.254167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2315.256989] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2315.284636] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2317.256058] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2319.281948] Lustre: Failing over lustre-OST0000 [ 2319.301106] LustreError: 160471:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2319.302714] LustreError: 160471:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2319.364049] Lustre: server umount lustre-OST0000 complete [ 2332.926335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2333.072756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2333.080136] Lustre: Skipped 1 previous similar message [ 2333.085905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2333.090647] Lustre: Skipped 1 previous similar message [ 2333.095882] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2333.954401] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2335.228945] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2336.160144] Lustre: 149977:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1755639185/real 1755639185] req@000000005a045865 x1840920992472576/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1755639201 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2336.797088] Lustre: Failing over lustre-MDT0000 [ 2336.870401] LustreError: 161439:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2336.872506] LustreError: 161439:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2336.905167] Lustre: server umount lustre-MDT0000 complete [ 2338.275673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2338.279801] Lustre: Skipped 2 previous similar messages [ 2341.344184] Lustre: 149977:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1755639190/real 1755639190] req@00000000bdce1f95 x1840920992474496/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1755639206 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2341.354570] Lustre: 149977:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2343.394444] LustreError: 158060:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.407274] LustreError: 158060:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 7 previous similar messages [ 2344.184089] LustreError: 158057:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2346.464181] Lustre: 149977:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1755639195/real 1755639195] req@00000000398e98c7 x1840920992475520/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1755639211 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2348.512961] LustreError: 158060:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2348.517108] LustreError: 158060:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2350.208278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2350.247120] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2350.250836] Lustre: 158053:0:(mgc_request.c:1876:mgc_process_log()) MGC192.168.204.132@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2350.321192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2350.323898] LustreError: Skipped 1 previous similar message [ 2350.337170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2350.368861] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2351.664785] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2354.423901] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2355.693752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2355.704811] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2355.712105] Lustre: Skipped 1 previous similar message [ 2355.745126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2360.807398] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2360.809438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2360.809919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2360.813666] Lustre: Skipped 1 previous similar message [ 2360.828898] LustreError: Skipped 1 previous similar message [ 2363.119598] LustreError: 162316:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2363.123150] LustreError: 162316:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2363.207769] Lustre: server umount lustre-OST0000 complete [ 2365.922511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2371.041637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2371.044069] Lustre: Skipped 4 previous similar messages [ 2379.232165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2379.269021] LustreError: 162518:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2379.272267] LustreError: 162518:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2379.294126] Lustre: server umount lustre-MDT0000 complete [ 2380.257112] LustreError: 158060:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.262254] LustreError: 158060:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2380.432635] LustreError: 158042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639245 with bad export cookie 17925060048029460397 [ 2380.436041] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2380.436423] LustreError: 158042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2380.597614] Lustre: server umount lustre-MDT0001 complete [ 2384.290396] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 2385.665258] Key type lgssc unregistered [ 2385.839574] LNet: 163246:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2385.844111] LNetError: 163246:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2385.852687] LNet: Removed LNI 192.168.204.132@tcp [ 2386.320763] Key type .llcrypt unregistered [ 2386.322927] Key type ._llcrypt unregistered [ 2409.074647] Key type ._llcrypt registered [ 2409.076481] Key type .llcrypt registered [ 2409.137452] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 2409.483509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2409.489525] alg: No test for adler32 (adler32-zlib) [ 2410.364626] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 2410.468252] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 2412.048156] Key type lgssc registered [ 2412.495834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2415.777703] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2415.783301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2416.960567] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2418.232740] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2421.792487] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2424.198108] LustreError: 164763:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2424.203772] LustreError: 164763:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2424.209276] LustreError: 164763:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 2424.303494] LustreError: 165280:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2424.398767] Lustre: server umount lustre-MDT0000 complete [ 2430.587528] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 2434.584727] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 2438.438931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2441.694869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2444.607640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2447.282876] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2449.991058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2450.016195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2450.131256] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2450.146264] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2450.205101] Lustre: lustre-MDT0000: new disk, initializing [ 2450.235602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2450.242585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2451.936443] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2457.045108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2457.067103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2457.102669] Lustre: 167909:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2457.139298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2457.141328] Lustre: Skipped 1 previous similar message [ 2457.184995] Lustre: lustre-MDT0001: new disk, initializing [ 2457.223728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2457.235978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2457.239143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2458.853224] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2462.507696] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2463.864997] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2467.176624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2467.215387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2467.337716] Lustre: lustre-OST0000: new disk, initializing [ 2467.339575] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2467.368296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2468.679840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2468.686176] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2468.731658] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2469.780835] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2473.624858] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2475.977933] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2476.802236] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 17:35:41 (1755639341) [ 2544.608537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2544.610740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2544.614193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2545.632668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2545.632838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2545.638554] Lustre: Skipped 2 previous similar messages [ 2550.752801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2550.756283] Lustre: Skipped 2 previous similar messages [ 2555.873045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2555.877056] Lustre: Skipped 2 previous similar messages [ 2556.384098] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2556.439549] LustreError: 171390:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2556.443211] LustreError: 171390:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2556.508654] Lustre: server umount lustre-MDT0000 complete [ 2557.887516] LustreError: 167901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639422 with bad export cookie 2318357973908435467 [ 2557.888630] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2557.893347] LustreError: 167901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2560.992698] LustreError: 167917:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2560.992746] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2561.003355] LustreError: 167917:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2561.008560] Lustre: Skipped 1 previous similar message [ 2561.008795] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2563.040405] LustreError: 170244:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.136921] LustreError: 170244:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2572.256304] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2572.256667] LustreError: 170244:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2572.256753] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2572.256760] Lustre: Skipped 2 previous similar messages [ 2572.306665] LustreError: 171593:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2572.309439] LustreError: 171593:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2572.388867] Lustre: server umount lustre-MDT0001 complete [ 2588.128131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2588.158547] LustreError: 171796:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2588.164043] LustreError: 171796:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2588.280485] Lustre: server umount lustre-OST0000 complete [ 2592.097776] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 2595.043222] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 2599.402890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2601.883712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2604.133958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2606.872779] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2609.860623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2609.892020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2609.990355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2610.005076] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2610.050392] Lustre: lustre-MDT0000: new disk, initializing [ 2610.075238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2610.081264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2611.494873] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2616.485856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2616.529799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2616.585101] Lustre: 174122:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2616.628699] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2616.632069] Lustre: Skipped 1 previous similar message [ 2616.689000] Lustre: lustre-MDT0001: new disk, initializing [ 2616.748703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2616.754904] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2618.359579] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2621.984142] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2623.423560] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2626.357344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2626.380241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2626.488900] Lustre: lustre-OST0000: new disk, initializing [ 2626.491452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2626.523958] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2626.526652] Lustre: Skipped 1 previous similar message [ 2628.635936] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2631.684279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2631.687676] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2631.730135] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2632.314482] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2634.158744] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2634.276606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2636.014433] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2636.151932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2641.897734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2641.898588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2641.902272] LustreError: Skipped 1 previous similar message [ 2641.903223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2641.909382] Lustre: Skipped 1 previous similar message [ 2643.182269] LustreError: 177037:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2643.184943] LustreError: 177037:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2643.259444] Lustre: server umount lustre-OST0000 complete [ 2647.012108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2647.019405] Lustre: Skipped 1 previous similar message [ 2659.296110] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2659.331941] LustreError: 177238:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2659.333838] LustreError: 177238:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2659.362216] Lustre: server umount lustre-MDT0000 complete [ 2660.579340] LustreError: 174897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639525 with bad export cookie 2318357973908594283 [ 2660.582290] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2660.582607] LustreError: 174897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2660.726292] Lustre: server umount lustre-MDT0001 complete [ 2664.186500] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:38:48 (1755639528) [ 2670.319415] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 2674.141358] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 2678.156315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2681.212628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2684.128648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2687.080704] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2689.588796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2689.607455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2689.682606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2689.692073] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2689.726456] Lustre: lustre-MDT0000: new disk, initializing [ 2689.748212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2689.753732] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2690.913734] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2695.279662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2695.302753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2695.332813] Lustre: 180418:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2695.425110] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2696.782678] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2699.775588] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2701.024427] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2703.879295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2703.919528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2706.756071] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2710.720094] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2713.101055] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2713.164125] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2723.301979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2723.306916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2723.310600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2723.314085] Lustre: Skipped 1 previous similar message [ 2723.317683] Lustre: Skipped 7 previous similar messages [ 2724.909116] LustreError: 182944:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2724.910717] LustreError: 182944:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2724.977798] Lustre: server umount lustre-OST0000 complete [ 2732.659924] Lustre: server umount lustre-MDT0000 complete [ 2733.538771] LustreError: 180428:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2733.545971] LustreError: 180428:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2734.228929] LustreError: 180408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639599 with bad export cookie 2318357973908595151 [ 2734.233792] LustreError: 180408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2734.234560] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2734.413174] Lustre: server umount lustre-MDT0001 complete [ 2737.795846] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:40:02 (1755639602) [ 2744.996111] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 2749.046535] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 2753.328840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2756.036402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2758.218069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2760.474106] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2763.318925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2763.342703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2763.422430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2763.433352] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2763.436046] Lustre: Skipped 3 previous similar messages [ 2763.481726] Lustre: lustre-MDT0000: new disk, initializing [ 2763.484358] Lustre: Skipped 2 previous similar messages [ 2763.508553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2763.510374] Lustre: Skipped 2 previous similar messages [ 2763.515863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2763.518630] Lustre: Skipped 2 previous similar messages [ 2765.290385] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2770.391801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2770.413175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2770.453653] Lustre: 186318:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2770.572117] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2772.357664] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2776.316484] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2778.107737] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2781.602474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2781.649823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2784.504396] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2788.274325] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2788.935170] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2799.074479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2799.075460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2799.078551] LustreError: Skipped 2 previous similar messages [ 2799.081609] Lustre: Skipped 2 previous similar messages [ 2799.082087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2799.086549] Lustre: Skipped 3 previous similar messages [ 2801.657153] LustreError: 188850:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2801.661083] LustreError: 188850:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2801.758726] Lustre: server umount lustre-OST0000 complete [ 2809.521268] Lustre: server umount lustre-MDT0000 complete [ 2811.311073] LustreError: 187100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639676 with bad export cookie 2318357973908596222 [ 2811.317247] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2811.317861] LustreError: 187100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2811.505129] Lustre: server umount lustre-MDT0001 complete [ 2816.292524] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 2817.854336] Key type lgssc unregistered [ 2817.988597] LNet: 189780:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2817.992348] LNetError: 189780:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2818.006464] LNet: Removed LNI 192.168.204.132@tcp [ 2818.483644] Key type .llcrypt unregistered [ 2818.486099] Key type ._llcrypt unregistered [ 2829.367686] Key type ._llcrypt registered [ 2829.368895] Key type .llcrypt registered [ 2829.411470] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 2829.720523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2829.758599] alg: No test for adler32 (adler32-zlib) [ 2830.608308] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 2830.709769] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 2832.304198] Key type lgssc registered [ 2832.666794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2835.753182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2835.765272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2836.987653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2838.425511] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2842.484525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2842.692182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2844.074594] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2846.473772] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2847.815350] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2850.024374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2850.175197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2852.771250] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2855.140618] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2861.490559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2876.899077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2876.903900] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2876.905820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2876.911366] Lustre: Skipped 1 previous similar message [ 2879.604125] LustreError: 193396:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2879.696575] Lustre: server umount lustre-OST0000 complete [ 2882.019264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2882.020502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2882.026435] Lustre: Skipped 1 previous similar message [ 2882.032948] Lustre: Skipped 2 previous similar messages [ 2887.137981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2887.141198] Lustre: Skipped 1 previous similar message [ 2887.624864] LustreError: 193598:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2887.628981] LustreError: 193598:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2887.672435] Lustre: server umount lustre-MDT0000 complete [ 2889.462770] LustreError: 191127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639754 with bad export cookie 231840730565088929 [ 2889.467126] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2889.467885] LustreError: 191127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2889.554253] LustreError: 193801:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2889.556602] LustreError: 193801:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2889.637239] Lustre: server umount lustre-MDT0001 complete [ 2894.049709] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 2895.220774] Key type lgssc unregistered [ 2895.373480] LNet: 194327:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2895.377309] LNetError: 194327:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2895.386631] LNet: Removed LNI 192.168.204.132@tcp [ 2895.858834] Key type .llcrypt unregistered [ 2895.860754] Key type ._llcrypt unregistered [ 2906.884334] Key type ._llcrypt registered [ 2906.885421] Key type .llcrypt registered [ 2906.926871] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 2907.209028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2907.248030] alg: No test for adler32 (adler32-zlib) [ 2908.103709] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 2908.182681] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 2909.760198] Key type lgssc registered [ 2910.310494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2913.935607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2913.947303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2915.147533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2916.817305] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2920.969397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2921.189958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2922.875828] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2925.375877] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2927.091970] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2930.258546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2930.427205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2932.972241] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2935.333146] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2940.652993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2945.797215] Lustre: lustre-OST0000: Not available for connect from 192.168.204.32@tcp (stopping) [ 2946.023933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2946.025211] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2946.031029] Lustre: Skipped 1 previous similar message [ 2948.913195] LustreError: 197928:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2949.009740] Lustre: server umount lustre-OST0000 complete [ 2951.153931] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff88a898f72800.ost_server_uuid 50 [ 2951.803571] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2962.099914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2962.100756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2962.106688] Lustre: Skipped 1 previous similar message [ 2962.109502] Lustre: Skipped 2 previous similar messages [ 2966.242855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2966.246461] Lustre: Skipped 2 previous similar messages [ 2968.271298] LustreError: 198276:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2968.275089] LustreError: 198276:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2968.313639] Lustre: server umount lustre-MDT0000 complete [ 2970.117124] LustreError: 195674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639835 with bad export cookie 4480054552121622205 [ 2970.122333] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2970.123947] LustreError: 195674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2970.224061] LustreError: 198479:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2970.227461] LustreError: 198479:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2970.327213] Lustre: server umount lustre-MDT0001 complete [ 2974.181266] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 17:43:58 (1755639838) [ 2977.294346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2977.508194] LustreError: 199150:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2977.517590] LustreError: 199150:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2977.530983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2979.229270] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2982.881716] LustreError: 199150:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2983.113618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2983.322727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2985.047790] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2987.532058] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2989.166567] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2992.297648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2992.462339] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2994.840875] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 2997.372701] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3000.582310] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3000.607906] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3000.686296] Lustre: lustre-OST0001: new disk, initializing [ 3000.689677] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3001.084156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3001.091527] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3001.136261] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3002.160484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3003.164346] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3007.100388] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3008.994903] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3008.999580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3009.009326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3012.584447] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3012.589251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3014.451244] LustreError: 202331:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3014.455130] LustreError: 202331:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3014.479640] Lustre: server umount lustre-OST0000 complete [ 3017.165512] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3017.275265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3017.465067] LustreError: 201823:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3019.095648] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3019.214828] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3022.586091] LustreError: 201823:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3022.595822] LustreError: 201823:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3027.704731] LustreError: 201823:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3027.715337] LustreError: 201823:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3032.034231] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3032.034554] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3032.039103] LustreError: Skipped 1 previous similar message [ 3032.047152] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3032.053498] Lustre: Skipped 4 previous similar messages [ 3035.439074] LustreError: 203019:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3035.440886] LustreError: 203019:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3035.531757] Lustre: server umount lustre-OST0001 complete [ 3043.298287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3043.301432] Lustre: Skipped 5 previous similar messages [ 3043.541512] LustreError: 203223:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3043.546559] LustreError: 203223:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3043.591077] Lustre: server umount lustre-MDT0000 complete [ 3044.936290] LustreError: 199130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639909 with bad export cookie 4480054552121623262 [ 3044.941173] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3044.942664] LustreError: 199130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3045.161840] Lustre: server umount lustre-MDT0001 complete [ 3047.867892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3049.746350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3051.416097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3053.282559] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3056.229530] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 17:45:20 (1755639920) [ 3059.518061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3059.684271] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3059.702668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3059.823408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3059.827203] Lustre: Skipped 1 previous similar message [ 3061.656373] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3066.012465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3066.072042] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3066.095273] Lustre: 204916:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3068.130385] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3070.666984] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3072.361390] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3075.579681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3075.688762] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3077.917442] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3079.795278] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3080.759640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3082.829856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3082.874813] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3085.446597] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3087.554663] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3090.913888] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3090.914033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3090.918430] LustreError: Skipped 1 previous similar message [ 3090.925516] Lustre: Skipped 3 previous similar messages [ 3090.929149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3090.934476] Lustre: Skipped 1 previous similar message [ 3094.451162] LustreError: 208089:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3094.455319] LustreError: 208089:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 3094.479896] Lustre: server umount lustre-OST0000 complete [ 3096.549336] LustreError: 206533:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3096.558766] LustreError: 206533:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 3106.791271] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3106.796128] LustreError: Skipped 1 previous similar message [ 3111.751047] Lustre: server umount lustre-OST0001 complete [ 3117.027284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3117.034551] Lustre: Skipped 3 previous similar messages [ 3119.700612] Lustre: server umount lustre-MDT0000 complete [ 3121.387061] LustreError: 205642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755639986 with bad export cookie 4480054552121624648 [ 3121.391762] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3121.393896] LustreError: 205642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3121.583763] Lustre: server umount lustre-MDT0001 complete [ 3124.280262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3126.178492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3127.860270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3129.540742] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3132.512258] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 17:46:37 (1755639997) [ 3139.266730] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 3142.720363] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 3147.081752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3150.154106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3152.964605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3155.833258] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3158.981103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3159.023726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3159.149171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3159.166657] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3159.220256] Lustre: lustre-MDT0000: new disk, initializing [ 3159.254729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3159.258304] Lustre: Skipped 3 previous similar messages [ 3159.268170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3160.897189] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3166.241230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3166.283351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3166.339414] Lustre: 212502:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3166.377700] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3166.380956] Lustre: Skipped 1 previous similar message [ 3166.422714] Lustre: lustre-MDT0001: new disk, initializing [ 3166.478874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3166.484857] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3168.025685] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3171.916453] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3173.486137] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3176.668552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3176.713099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3176.833311] Lustre: lustre-OST0000: new disk, initializing [ 3176.835911] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3179.307949] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3183.335880] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3185.164919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3185.170961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3185.174156] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3185.218897] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3186.335295] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3187.948961] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3188.049208] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3190.245233] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3190.245437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3190.249795] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3190.249805] Lustre: Skipped 7 previous similar messages [ 3190.251155] LustreError: Skipped 1 previous similar message [ 3190.258583] Lustre: Skipped 1 previous similar message [ 3190.957890] LustreError: 215419:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3190.959654] LustreError: 215419:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3191.017200] Lustre: server umount lustre-OST0000 complete [ 3207.136122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3207.216850] Lustre: server umount lustre-MDT0000 complete [ 3208.972101] LustreError: 212493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640073 with bad export cookie 4480054552121626209 [ 3208.978850] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3208.978907] LustreError: 212493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3209.162100] Lustre: server umount lustre-MDT0001 complete [ 3213.249710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3213.491846] LustreError: 216298:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3213.499805] LustreError: 216298:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 3215.238697] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3219.008739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3220.616185] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3222.992196] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3224.806212] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3228.076248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3228.223196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3228.226104] Lustre: Skipped 4 previous similar messages [ 3230.857397] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3233.405371] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3235.172973] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3239.432619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3241.183996] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3241.305975] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3244.004632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3244.360407] Lustre: server umount lustre-OST0000 complete [ 3246.794435] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3246.876643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3248.513939] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3248.637967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3273.056345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3275.347950] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3277.767500] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3278.622509] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3280.016976] LustreError: 220693:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3280.020900] LustreError: 220693:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3280.107828] Lustre: server umount lustre-OST0000 complete [ 3283.427201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3283.428158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3283.434325] Lustre: Skipped 5 previous similar messages [ 3283.437988] Lustre: Skipped 9 previous similar messages [ 3288.049770] Lustre: server umount lustre-MDT0000 complete [ 3289.261017] LustreError: 216277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640154 with bad export cookie 4480054552121627091 [ 3289.263891] LustreError: 216277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3289.264549] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3289.392371] Lustre: server umount lustre-MDT0001 complete [ 3291.875792] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 17:49:16 (1755640156) [ 3293.804605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3293.940267] LustreError: 221766:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3293.945198] LustreError: 221766:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3295.074379] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3297.643443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3298.894574] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3300.516539] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3301.653544] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3303.688704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3305.475251] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3307.144330] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3308.362296] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3308.433486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3309.630384] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3309.710991] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3311.994023] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3312.034472] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3312.088887] Lustre: lustre-OST0001: new disk, initializing [ 3312.090619] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3313.725303] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3315.388238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3315.391532] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3315.415675] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3316.578334] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3317.801378] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3317.861897] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3319.134140] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3319.194407] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3320.805367] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3320.808246] LustreError: Skipped 1 previous similar message [ 3321.850940] Lustre: server umount lustre-OST0001 complete [ 3324.317221] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3324.435145] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3326.072633] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3326.183317] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3350.964404] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3353.661219] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3356.143757] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3356.155437] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to (at 0@lo) [ 3356.156233] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3356.357677] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3357.230238] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3364.290150] Lustre: server umount lustre-OST0001 complete [ 3366.377183] LustreError: 223351:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3366.387578] LustreError: 223351:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 16 previous similar messages [ 3383.177895] Lustre: server umount lustre-OST0000 complete [ 3391.217277] Lustre: server umount lustre-MDT0000 complete [ 3392.968262] LustreError: 221746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640257 with bad export cookie 4480054552121628022 [ 3392.974862] LustreError: 221746:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3392.975193] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3407.328157] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3407.449422] Lustre: server umount lustre-MDT0001 complete [ 3410.174178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3412.042356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3413.877837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3415.749793] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3419.130805] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 17:51:23 (1755640283) [ 3422.298051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3422.445440] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3422.464611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3422.580142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3422.584111] Lustre: Skipped 6 previous similar messages [ 3424.409970] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3428.500660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3428.553247] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3428.571568] Lustre: 229674:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3430.585618] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3433.229807] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3434.908658] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3438.127110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3438.191599] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3440.379652] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3442.519982] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3447.293089] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3447.327480] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3449.981708] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3452.680700] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3453.415227] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3454.928736] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3454.990573] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3456.365524] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3456.473348] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3458.176368] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88a8857c9800.ost_server_uuid 50 [ 3458.894773] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff88a8857c9800.ost_server_uuid in FULL state after 0 sec [ 3459.414201] Lustre: Permanently deactivating lustre-OST0001 [ 3459.425406] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3460.462403] Lustre: Permanently reactivating lustre-OST0001 [ 3464.165693] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3464.167870] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3464.170916] LustreError: Skipped 5 previous similar messages [ 3464.178048] Lustre: Skipped 8 previous similar messages [ 3464.179987] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3464.186977] Lustre: Skipped 13 previous similar messages [ 3467.825137] LustreError: 233623:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3467.829177] LustreError: 233623:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 3467.853645] Lustre: server umount lustre-OST0001 complete [ 3475.786708] Lustre: server umount lustre-OST0000 complete [ 3483.768837] Lustre: server umount lustre-MDT0000 complete [ 3485.579909] LustreError: 233471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640350 with bad export cookie 4480054552121629296 [ 3485.587275] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3485.589627] LustreError: 233471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3485.769394] Lustre: server umount lustre-MDT0001 complete [ 3488.262927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3490.107325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3491.975457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3493.836503] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3497.147739] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 17:52:41 (1755640361) [ 3497.637194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3500.864894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3501.021851] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3501.038866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3501.044756] Lustre: Skipped 5 previous similar messages [ 3502.910662] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3506.985728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3507.045127] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3507.062338] Lustre: 235779:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3508.856354] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3511.497923] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3513.230471] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3516.501359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3516.543583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3516.650245] Lustre: Permanently deactivating lustre-OST0000 [ 3516.656212] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3518.470882] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3518.475924] LustreError: 237614:0:(osp_precreate.c:654:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 3518.482755] LustreError: 237614:0:(osp_precreate.c:1358:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 3519.353875] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3522.002436] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3525.339633] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3528.039170] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3530.721302] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3531.849275] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3536.729975] Lustre: Permanently reactivating lustre-OST0000 [ 3536.739777] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 3536.743776] Lustre: Skipped 2 previous similar messages [ 3536.967201] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3536.970609] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3536.977742] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.204.132@tcp (at 0@lo) [ 3536.980815] Lustre: Skipped 1 previous similar message [ 3536.995562] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3537.003690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3550.407317] Lustre: server umount lustre-OST0001 complete [ 3552.230639] LustreError: 238862:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3552.239958] LustreError: 238862:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 15 previous similar messages [ 3558.344536] Lustre: server umount lustre-OST0000 complete [ 3566.381715] Lustre: server umount lustre-MDT0000 complete [ 3568.129832] LustreError: 235769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640433 with bad export cookie 4480054552121630927 [ 3568.135824] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3568.136795] LustreError: 235769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3568.331350] Lustre: server umount lustre-MDT0001 complete [ 3573.496434] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 3575.070355] Key type lgssc unregistered [ 3575.245746] LNet: 240150:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3575.250388] LNetError: 240150:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3575.260639] LNet: Removed LNI 192.168.204.132@tcp [ 3575.736406] Key type .llcrypt unregistered [ 3575.737726] Key type ._llcrypt unregistered [ 3586.548621] Key type ._llcrypt registered [ 3586.550216] Key type .llcrypt registered [ 3586.603597] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 3586.970269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3587.020288] alg: No test for adler32 (adler32-zlib) [ 3587.879703] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 3587.965988] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 3589.544149] Key type lgssc registered [ 3590.136358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3591.762510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3594.968770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3594.981479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3596.231457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3597.911383] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3602.073956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3602.116774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3602.173871] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3602.192362] Lustre: 241557:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3602.201072] Lustre: Permanently deactivating lustre-MDT0001 [ 3602.210880] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3602.394969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3603.954348] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3603.958121] LustreError: 241598:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3603.963253] LustreError: 241598:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3603.968006] LustreError: 241598:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 1, failed: rc = -108 [ 3604.034427] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3606.584468] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3608.406935] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3611.639863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3611.827187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3612.838883] LustreError: 243310:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3612.845131] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3612.848492] LustreError: 243310:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3614.306108] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3616.803857] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3618.276034] LustreError: 243173:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3620.154220] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3620.286678] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3620.927155] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3622.001770] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3622.011182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3622.920313] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3625.570368] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3627.474850] Lustre: Permanently reactivating lustre-MDT0001 [ 3627.485091] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3627.489958] Lustre: Skipped 2 previous similar messages [ 3631.876862] Lustre: lustre-MDT0001: Client 1385e60a-47e1-4262-a4a4-e251319088a7 (at 192.168.204.32@tcp) reconnecting [ 3634.608940] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3635.404597] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3637.587531] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3637.683434] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3639.436638] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3639.559798] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3640.816605] Lustre: Permanently deactivating lustre-MDT0001 [ 3640.827072] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3640.831086] Lustre: Skipped 2 previous similar messages [ 3647.514373] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3653.601373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3653.603251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3653.606887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3655.142315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3655.144132] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3655.145852] Lustre: Skipped 1 previous similar message [ 3655.153339] Lustre: Skipped 1 previous similar message [ 3659.666792] LustreError: 245615:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3659.723365] Lustre: server umount lustre-MDT0000 complete [ 3660.259694] LustreError: 241568:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3660.270441] LustreError: 241568:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3661.587370] LustreError: 241547:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640526 with bad export cookie 1726579671688767514 [ 3661.597296] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3661.688760] LustreError: 245818:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3661.692380] LustreError: 245818:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3661.813997] Lustre: server umount lustre-MDT0001 complete [ 3669.683572] LustreError: 246018:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3669.687468] LustreError: 246018:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3669.710024] Lustre: server umount lustre-OST0000 complete [ 3675.194572] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 17:55:39 (1755640539) [ 3680.384790] LustreError: 247022:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3680.388490] LustreError: 247022:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3680.500588] Lustre: server umount lustre-OST0001 complete [ 3683.351764] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 3687.849896] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 3692.238245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3695.475859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3698.268757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3701.209576] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3704.225598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3704.263609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3704.398963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3704.405401] Lustre: Skipped 2 previous similar messages [ 3704.420434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3704.488131] Lustre: lustre-MDT0000: new disk, initializing [ 3704.535415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3704.547267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3706.287998] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3711.926114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3711.961634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3712.001968] Lustre: 249195:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3712.038635] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3712.041949] Lustre: Skipped 1 previous similar message [ 3712.084193] Lustre: lustre-MDT0001: new disk, initializing [ 3712.144143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3712.149404] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3713.723992] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3717.456423] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3719.117911] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3722.288148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3722.331078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3722.485670] Lustre: lustre-OST0000: new disk, initializing [ 3722.489244] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3722.533737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3722.537502] Lustre: Skipped 1 previous similar message [ 3725.025422] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3729.051213] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3730.441922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3730.452636] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3730.498547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3735.612809] LustreError: 249201:0:(mdt_handler.c:3249:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3740.854651] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3740.893380] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3740.985924] Lustre: lustre-OST0001: new disk, initializing [ 3740.989605] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3741.041238] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3743.605254] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3745.704142] LustreError: 249201:0:(mdt_handler.c:3249:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3745.717841] LustreError: 249201:0:(mdt_handler.c:3249:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3746.829792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3746.836083] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3746.887650] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3747.657374] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3748.530854] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3755.808195] LustreError: 249201:0:(mdt_handler.c:3249:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3756.807932] Lustre: lustre-OST0001: Not available for connect from 192.168.204.32@tcp (stopping) [ 3757.029921] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3757.034537] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3757.041757] Lustre: Skipped 2 previous similar messages [ 3761.914493] Lustre: lustre-OST0001: Not available for connect from 192.168.204.32@tcp (stopping) [ 3761.919199] Lustre: Skipped 2 previous similar messages [ 3767.034269] Lustre: lustre-OST0001: Not available for connect from 192.168.204.32@tcp (stopping) [ 3767.038942] Lustre: Skipped 2 previous similar messages [ 3770.848175] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3770.876855] LustreError: 252668:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3770.880743] LustreError: 252668:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3770.910437] Lustre: server umount lustre-OST0001 complete [ 3772.153177] LustreError: 250914:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3777.514298] LustreError: 251530:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3777.524563] LustreError: 251530:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3786.722207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3786.722273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3786.726901] LustreError: Skipped 2 previous similar messages [ 3786.735055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3786.740581] Lustre: Skipped 3 previous similar messages [ 3787.752274] LustreError: 252787:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3787.761433] LustreError: 252787:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3789.619195] LustreError: 252895:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3789.622972] LustreError: 252895:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3789.712955] Lustre: server umount lustre-OST0000 complete [ 3792.867493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3792.874777] Lustre: Skipped 2 previous similar messages [ 3797.811993] Lustre: server umount lustre-MDT0000 complete [ 3799.567085] LustreError: 249187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640664 with bad export cookie 1726579671688770293 [ 3799.573859] LustreError: 249187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3799.574106] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3805.708678] LustreError: 253298:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3805.712519] LustreError: 253298:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3805.808274] Lustre: server umount lustre-MDT0001 complete [ 3810.851813] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 3812.349002] Key type lgssc unregistered [ 3812.515527] LNet: 253825:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3812.519528] LNetError: 253825:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3812.528530] LNet: Removed LNI 192.168.204.132@tcp [ 3812.910353] Key type .llcrypt unregistered [ 3812.912018] Key type ._llcrypt unregistered [ 3827.991423] Key type ._llcrypt registered [ 3827.992633] Key type .llcrypt registered [ 3828.040406] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 3836.301605] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 3836.584691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3836.592123] alg: No test for adler32 (adler32-zlib) [ 3837.466780] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 3837.572770] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 3839.176282] Key type lgssc registered [ 3839.685499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3844.346638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3847.822330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3850.904512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3854.265743] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3857.607505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3857.640311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3857.650773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3858.768755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3858.786924] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3858.831349] Lustre: lustre-MDT0000: new disk, initializing [ 3858.865758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3858.876729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3860.583119] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3866.156939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3866.196068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3866.251454] Lustre: 257739:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3866.289106] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3866.292670] Lustre: Skipped 1 previous similar message [ 3866.336037] Lustre: lustre-MDT0001: new disk, initializing [ 3866.382077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3866.396803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3866.401229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3867.925355] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3872.065460] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3873.912980] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3877.202966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3877.224543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3877.308572] Lustre: lustre-OST0000: new disk, initializing [ 3877.310679] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3877.338141] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3879.576591] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3883.019440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3883.026172] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3883.083252] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3883.690654] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3885.568708] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3885.698298] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3887.638123] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3887.764375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3893.218355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3893.218548] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3893.220338] LustreError: Skipped 1 previous similar message [ 3893.220701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3893.223214] Lustre: Skipped 1 previous similar message [ 3894.771258] LustreError: 260647:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3894.867578] Lustre: server umount lustre-OST0000 complete [ 3897.313189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3897.315968] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3897.320112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3897.321919] Lustre: Skipped 1 previous similar message [ 3898.338109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3898.339655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3902.539574] LustreError: 260848:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3902.543564] LustreError: 260848:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3902.579894] Lustre: server umount lustre-MDT0000 complete [ 3904.413784] LustreError: 257730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640769 with bad export cookie 16518692278445891260 [ 3904.419418] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3904.420712] LustreError: 257730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3904.519282] LustreError: 261050:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3904.522364] LustreError: 261050:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3904.608102] Lustre: server umount lustre-MDT0001 complete [ 3908.818200] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 17:59:33 (1755640773) [ 3912.018811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3912.214595] LustreError: 261722:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3912.222689] LustreError: 261722:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3912.249514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3913.891376] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3917.282103] LustreError: 261722:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3917.845818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3918.051177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3919.744716] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3922.409928] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3924.276286] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3927.554602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3927.734698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3930.211060] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 3932.906650] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3984.359211] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3984.359597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3984.363746] LustreError: Skipped 1 previous similar message [ 3984.372447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3984.376946] Lustre: Skipped 2 previous similar messages [ 3988.081162] LustreError: 264033:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3988.085211] LustreError: 264033:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3988.210211] Lustre: server umount lustre-OST0000 complete [ 3990.294626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3996.104761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3996.275331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3998.800258] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4001.292745] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4004.455159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4004.462741] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to (at 0@lo) [ 4004.462803] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4035.556749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4035.558407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4035.559523] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4035.572153] Lustre: Skipped 2 previous similar messages [ 4041.264733] LustreError: 265701:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4041.270528] LustreError: 265701:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4041.390709] Lustre: server umount lustre-OST0000 complete [ 4045.795283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4045.803978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4045.807991] Lustre: Skipped 4 previous similar messages [ 4049.262996] Lustre: server umount lustre-MDT0000 complete [ 4050.913778] LustreError: 262347:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4050.925189] LustreError: 262347:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4051.045762] LustreError: 261703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640915 with bad export cookie 16518692278445892135 [ 4051.051817] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4051.052664] LustreError: 261703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4051.145894] LustreError: 266103:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4051.149340] LustreError: 266103:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4051.248344] Lustre: server umount lustre-MDT0001 complete [ 4056.297741] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 4057.787115] Key type lgssc unregistered [ 4057.974786] LNet: 266629:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4057.979419] LNetError: 266629:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4057.987563] LNet: Removed LNI 192.168.204.132@tcp [ 4058.374828] Key type .llcrypt unregistered [ 4058.376897] Key type ._llcrypt unregistered [ 4070.361378] Key type ._llcrypt registered [ 4070.363062] Key type .llcrypt registered [ 4070.400133] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 4070.744791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4070.789060] alg: No test for adler32 (adler32-zlib) [ 4071.632204] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 4071.719881] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 4073.296111] Key type lgssc registered [ 4073.734234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4077.342480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4077.354363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4078.585733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4080.213887] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4084.214276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4084.413590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4086.196639] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4088.828320] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4090.659302] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4094.024692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4094.198639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4096.959428] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4099.606552] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4103.411345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4115.937539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4115.942099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4115.950277] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4117.107157] LustreError: 270999:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4117.209522] Lustre: server umount lustre-OST0000 complete [ 4120.545495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4120.549856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4120.557911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4124.130718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4124.132089] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4128.230987] LustreError: 271201:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4128.235058] LustreError: 271201:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4128.273310] Lustre: server umount lustre-MDT0000 complete [ 4129.250361] LustreError: 267998:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4129.259849] LustreError: 267998:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4130.117821] LustreError: 267979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755640995 with bad export cookie 7504813660473838697 [ 4130.124562] LustreError: 267979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4130.126575] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4130.224049] LustreError: 271402:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4130.227720] LustreError: 271402:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4130.361777] Lustre: server umount lustre-MDT0001 complete [ 4135.702685] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 4137.130504] Key type lgssc unregistered [ 4137.298732] LNet: 271929:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4137.301855] LNetError: 271929:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4137.315845] LNet: Removed LNI 192.168.204.132@tcp [ 4137.833320] Key type .llcrypt unregistered [ 4137.835441] Key type ._llcrypt unregistered [ 4147.014788] Key type ._llcrypt registered [ 4147.016635] Key type .llcrypt registered [ 4147.082855] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 4147.424444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4147.433918] alg: No test for adler32 (adler32-zlib) [ 4148.289615] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 4148.376908] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 4149.976227] Key type lgssc registered [ 4150.516512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4154.564426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4154.576967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4155.819116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4157.702436] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4161.918657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4162.136838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4163.913501] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4166.498492] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4168.334383] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4171.691773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4171.860180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4174.567707] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4177.176012] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4181.102125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4194.788037] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4194.792598] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.800701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4196.838759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4196.840439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4196.842360] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4198.130159] LustreError: 275507:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4198.229339] Lustre: server umount lustre-OST0000 complete [ 4201.956908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4201.957918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4201.964231] Lustre: Skipped 1 previous similar message [ 4201.969940] Lustre: Skipped 2 previous similar messages [ 4206.222274] LustreError: 275709:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4206.225874] LustreError: 275709:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4206.261957] Lustre: server umount lustre-MDT0000 complete [ 4207.075258] LustreError: 273816:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4207.084527] LustreError: 273816:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4208.012749] LustreError: 273090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641072 with bad export cookie 2134780320771019258 [ 4208.019592] LustreError: 273090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4208.019853] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4208.105877] LustreError: 275910:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4208.108500] LustreError: 275910:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4208.219132] Lustre: server umount lustre-MDT0001 complete [ 4213.317286] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 4214.963290] Key type lgssc unregistered [ 4215.164630] LNet: 276437:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4215.169363] LNetError: 276437:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4215.182594] LNet: Removed LNI 192.168.204.132@tcp [ 4215.679683] Key type .llcrypt unregistered [ 4215.681900] Key type ._llcrypt unregistered [ 4227.888436] Key type ._llcrypt registered [ 4227.890367] Key type .llcrypt registered [ 4227.945271] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 4228.334668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4228.398079] alg: No test for adler32 (adler32-zlib) [ 4229.283511] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 4229.404749] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 4231.016176] Key type lgssc registered [ 4231.598257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4235.337403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4235.349773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4236.567580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4238.475352] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4242.581034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4242.766077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4244.498380] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4247.173160] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4248.909465] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4252.246979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4252.428070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4254.986124] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4257.641289] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4261.678207] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4274.145524] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4274.151667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4274.164597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4277.221882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4277.222677] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4277.232787] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4278.772924] LustreError: 280807:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4278.860664] Lustre: server umount lustre-OST0000 complete [ 4282.338984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4282.347664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4282.351511] Lustre: Skipped 2 previous similar messages [ 4286.669441] LustreError: 281009:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4286.674234] LustreError: 281009:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4286.709654] Lustre: server umount lustre-MDT0000 complete [ 4287.458828] LustreError: 277807:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4287.468235] LustreError: 277807:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4288.409895] LustreError: 277787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641153 with bad export cookie 16158374508082033457 [ 4288.416756] LustreError: 277787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4288.417164] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4288.501587] LustreError: 281212:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4288.504214] LustreError: 281212:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4288.582619] Lustre: server umount lustre-MDT0001 complete [ 4293.337188] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 4294.833721] Key type lgssc unregistered [ 4295.031682] LNet: 281738:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4295.036280] LNetError: 281738:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4295.044622] LNet: Removed LNI 192.168.204.132@tcp [ 4295.518293] Key type .llcrypt unregistered [ 4295.520052] Key type ._llcrypt unregistered [ 4303.869345] Key type ._llcrypt registered [ 4303.870970] Key type .llcrypt registered [ 4303.914960] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 4304.233533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4304.304313] alg: No test for adler32 (adler32-zlib) [ 4305.190107] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 4305.307080] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 4306.912174] Key type lgssc registered [ 4307.567179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4311.582665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4311.596169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4312.832865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4314.550989] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4318.895081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4319.107943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4320.963363] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4323.629430] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4325.428120] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4328.710731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4328.898105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4331.667852] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4334.261407] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4337.138909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4351.972382] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4351.977042] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4351.985282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4353.509497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4353.513885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4353.514240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4353.524280] Lustre: Skipped 1 previous similar message [ 4355.250105] LustreError: 285343:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4355.352250] Lustre: server umount lustre-OST0000 complete [ 4358.629211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4358.631203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4358.636162] Lustre: Skipped 1 previous similar message [ 4363.342828] LustreError: 285544:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4363.346505] LustreError: 285544:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4363.389938] Lustre: server umount lustre-MDT0000 complete [ 4363.747738] LustreError: 282939:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4363.757287] LustreError: 282939:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4365.233395] LustreError: 282892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641230 with bad export cookie 10842141166415396765 [ 4365.240396] LustreError: 282892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4365.240889] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4365.351583] LustreError: 285745:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4365.355291] LustreError: 285745:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4365.482378] Lustre: server umount lustre-MDT0001 complete [ 4370.392922] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 4372.005454] Key type lgssc unregistered [ 4372.174668] LNet: 286271:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4372.179735] LNetError: 286271:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4372.188753] LNet: Removed LNI 192.168.204.132@tcp [ 4372.705314] Key type .llcrypt unregistered [ 4372.707351] Key type ._llcrypt unregistered [ 4377.438342] Key type ._llcrypt registered [ 4377.439898] Key type .llcrypt registered [ 4377.497735] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4385.599365] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 4393.501572] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 4393.865919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4393.873983] alg: No test for adler32 (adler32-zlib) [ 4394.755659] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 4394.873265] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 4396.488182] Key type lgssc registered [ 4397.121140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4401.774836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4405.188340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4408.277091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4411.461273] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4414.757469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4414.796068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4414.806192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4415.934374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4415.946220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4415.999592] Lustre: lustre-MDT0000: new disk, initializing [ 4416.035643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4416.046093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4417.932371] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4423.496560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4423.540655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4423.601850] Lustre: 289753:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4423.651419] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4423.654951] Lustre: Skipped 1 previous similar message [ 4423.706042] Lustre: lustre-MDT0001: new disk, initializing [ 4423.765974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4423.789884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4423.796938] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4425.671818] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4429.597091] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4431.289689] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4434.561567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4434.600281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4434.767731] Lustre: lustre-OST0000: new disk, initializing [ 4434.771183] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4434.822210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4437.531523] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4441.535120] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4442.649832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4442.655921] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4442.709839] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4443.402928] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4443.534294] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4445.403527] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4445.536458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4447.722349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4447.722525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4447.731427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4447.734661] LustreError: Skipped 1 previous similar message [ 4447.738236] Lustre: Skipped 1 previous similar message [ 4452.593163] LustreError: 292659:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4452.691498] Lustre: server umount lustre-OST0000 complete [ 4457.955207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4457.956175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4457.963059] Lustre: Skipped 2 previous similar messages [ 4460.620790] LustreError: 292861:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4460.625045] LustreError: 292861:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4460.664899] Lustre: server umount lustre-MDT0000 complete [ 4462.451113] LustreError: 291486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641327 with bad export cookie 17117692489556829594 [ 4462.457487] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4462.458255] LustreError: 291486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4462.539571] LustreError: 293064:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4462.542165] LustreError: 293064:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4462.646408] Lustre: server umount lustre-MDT0001 complete [ 4466.809499] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 18:08:51 (1755641331) [ 4469.979020] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4470.204457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4471.845222] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4475.361974] LustreError: 293734:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4475.371240] LustreError: 293734:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4475.923616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4476.113215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4477.760538] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4480.342402] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4482.141640] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4485.444287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4485.608122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4488.304316] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4490.823401] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4506.086745] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4506.086811] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4506.091331] LustreError: Skipped 1 previous similar message [ 4506.092290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4506.099937] Lustre: Skipped 1 previous similar message [ 4506.108672] Lustre: Skipped 2 previous similar messages [ 4510.257311] LustreError: 295986:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4510.261167] LustreError: 295986:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4510.360151] Lustre: server umount lustre-OST0000 complete [ 4512.224693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4512.227311] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4512.231761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4514.321288] LustreError: 296187:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4514.325398] LustreError: 296187:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4514.363122] Lustre: server umount lustre-MDT0000 complete [ 4516.100380] LustreError: 293715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641380 with bad export cookie 17117692489556830462 [ 4516.107908] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4516.108399] LustreError: 293715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4516.296865] Lustre: server umount lustre-MDT0001 complete [ 4521.411351] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 4522.948099] Key type lgssc unregistered [ 4523.130742] LNet: 296915:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4523.135438] LNetError: 296915:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4523.149591] LNet: Removed LNI 192.168.204.132@tcp [ 4523.629654] Key type .llcrypt unregistered [ 4523.631979] Key type ._llcrypt unregistered [ 4535.583085] Key type ._llcrypt registered [ 4535.585247] Key type .llcrypt registered [ 4535.644884] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 4543.087878] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 4543.374512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4543.383252] alg: No test for adler32 (adler32-zlib) [ 4544.266020] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 4544.361612] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 4545.952100] Key type lgssc registered [ 4546.404917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4550.625941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4553.597699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4555.977981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4559.147578] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4561.275879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4563.958255] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4566.759901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4566.792531] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4566.804265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4567.914064] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4567.932720] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4567.984079] Lustre: lustre-MDT0000: new disk, initializing [ 4568.025971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4568.035768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4569.921876] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4575.422877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4575.466859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4575.525849] Lustre: 300680:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4575.574332] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4575.577855] Lustre: Skipped 1 previous similar message [ 4575.630571] Lustre: lustre-MDT0001: new disk, initializing [ 4575.685411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4575.706887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4575.713500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4576.976504] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4581.002585] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4582.305743] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4585.607423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4585.656393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4585.818464] Lustre: lustre-OST2710: new disk, initializing [ 4585.821988] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4585.879220] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4588.043382] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4591.482621] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4594.276643] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4594.309541] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4594.396049] Lustre: lustre-OST03e8: new disk, initializing [ 4594.399559] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4594.457799] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4595.635644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4595.644154] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4595.718805] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4595.721224] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4596.762570] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4599.045140] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4601.943079] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4602.003154] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4603.271409] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4603.346661] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4604.868377] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4604.993436] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4606.669000] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4606.728599] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4611.048033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4611.050245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4611.055354] Lustre: Skipped 3 previous similar messages [ 4611.059091] Lustre: Skipped 2 previous similar messages [ 4613.837410] LustreError: 304971:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 4613.876201] Lustre: server umount lustre-MDT0000 complete [ 4615.588675] LustreError: 302416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641480 with bad export cookie 4007309002227527560 [ 4615.594532] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4615.595611] LustreError: 302416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4615.688312] LustreError: 305173:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4615.690984] LustreError: 305173:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4615.787582] Lustre: server umount lustre-MDT0001 complete [ 4627.507090] LustreError: 305373:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4627.508964] LustreError: 305373:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4627.525816] Lustre: server umount lustre-OST2710 complete [ 4639.415648] LustreError: 305575:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4639.419788] LustreError: 305575:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4639.527138] Lustre: server umount lustre-OST03e8 complete [ 4647.253969] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 4651.014354] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 4655.068263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4658.131203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4660.971704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4663.747742] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4667.048188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4667.077635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4667.201505] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4667.221169] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4667.276337] Lustre: lustre-MDT0000: new disk, initializing [ 4667.315366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4667.327252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4667.332275] Lustre: Skipped 1 previous similar message [ 4669.158985] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4674.552711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4674.593131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4674.643669] Lustre: 308356:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4674.785059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4674.790580] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4674.795408] Lustre: Skipped 1 previous similar message [ 4676.471429] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4680.363352] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4681.973095] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4685.255205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4685.283810] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4685.386555] Lustre: lustre-OST0000: new disk, initializing [ 4685.388510] Lustre: Skipped 1 previous similar message [ 4685.391486] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4685.394988] Lustre: Skipped 2 previous similar messages [ 4685.429414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4685.432297] Lustre: Skipped 1 previous similar message [ 4687.888638] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4691.715357] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4691.981940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4691.988074] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4692.050739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4693.557599] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4693.672505] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4695.422776] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4695.546284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4697.061790] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4697.063403] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4697.070232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4697.073382] LustreError: Skipped 1 previous similar message [ 4698.480176] LustreError: 311267:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4698.483988] LustreError: 311267:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4698.580968] Lustre: server umount lustre-OST0000 complete [ 4700.641484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4700.646067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4700.653068] Lustre: Skipped 1 previous similar message [ 4700.656309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4700.659882] Lustre: Skipped 1 previous similar message [ 4706.508451] LustreError: 311468:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4706.512321] LustreError: 311468:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4706.549331] Lustre: server umount lustre-MDT0000 complete [ 4708.280312] LustreError: 308349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641573 with bad export cookie 4007309002227529121 [ 4708.286868] LustreError: 308349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4708.286979] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4708.472646] Lustre: server umount lustre-MDT0001 complete [ 4712.317463] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 18:12:57 (1755641577) [ 4713.048699] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4713.865858] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 18:12:58 (1755641578) [ 4715.100635] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 4718.464244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4720.111715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4721.712088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4722.831760] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4723.262267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4726.395881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4726.514594] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4726.525385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4726.588479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4728.209753] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4732.172806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4732.230155] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4732.247088] Lustre: 313791:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4733.964331] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4736.575688] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4738.007971] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4740.927167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4740.970247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4741.072984] Lustre: Denying initial registration attempt from nid 192.168.204.132@tcp, specified as failover [ 4741.077782] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4741.081712] LustreError: 315398:0:(tgt_mount.c:2419:server_fill_super()) Unable to start targets: -99 [ 4741.084827] LustreError: 315398:0:(tgt_mount.c:1913:server_put_super()) no obd lustre-OST0000 [ 4741.087530] LustreError: 315398:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4741.153818] Lustre: server umount lustre-OST0000 complete [ 4741.156279] LustreError: 315398:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4746.724261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4746.725505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4746.731550] Lustre: Skipped 2 previous similar messages [ 4746.735131] Lustre: Skipped 2 previous similar messages [ 4750.037168] LustreError: 315766:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4750.041076] LustreError: 315766:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4750.083158] Lustre: server umount lustre-MDT0000 complete [ 4751.841859] LustreError: 313803:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4751.851185] LustreError: 313803:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 4751.927977] LustreError: 314524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641616 with bad export cookie 4007309002227530010 [ 4751.936457] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4752.099328] Lustre: server umount lustre-MDT0001 complete [ 4757.171804] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 4758.837694] Key type lgssc unregistered [ 4759.021701] LNet: 316495:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4759.026433] LNetError: 316495:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4759.038696] LNet: Removed LNI 192.168.204.132@tcp [ 4759.541535] Key type .llcrypt unregistered [ 4759.543415] Key type ._llcrypt unregistered [ 4764.156203] Key type ._llcrypt registered [ 4764.157600] Key type .llcrypt registered [ 4764.214410] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 4764.635286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4764.670777] alg: No test for adler32 (adler32-zlib) [ 4765.586230] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 4765.705069] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 4767.304149] Key type lgssc registered [ 4767.940126] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4771.528460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4773.420665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4775.152840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4777.017955] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4777.467329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4785.924806] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 4789.983099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4789.993556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4791.117770] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4791.130065] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4791.218568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4792.998074] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4797.090948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4797.147441] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4797.165726] Lustre: 319215:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4797.322314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4798.984879] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4801.512526] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4803.392624] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4806.692081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4806.739078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4806.851607] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4806.963267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4809.635917] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4812.304126] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4813.793481] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4813.797800] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4813.805543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4814.309347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4814.310783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4814.313791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4815.221156] LustreError: 321480:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4815.314614] Lustre: server umount lustre-OST0000 complete [ 4817.889463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4817.894056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4817.902040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4817.905637] Lustre: Skipped 1 previous similar message [ 4823.180440] LustreError: 321682:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4823.184409] LustreError: 321682:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4823.219964] Lustre: server umount lustre-MDT0000 complete [ 4824.981912] LustreError: 319207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641689 with bad export cookie 3916165840155124277 [ 4824.988671] LustreError: 319207:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4824.988890] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4825.070259] LustreError: 321883:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4825.073743] LustreError: 321883:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4825.164946] Lustre: server umount lustre-MDT0001 complete [ 4829.582072] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 4831.000460] Key type lgssc unregistered [ 4831.157714] LNet: 322410:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4831.162726] LNetError: 322410:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4831.172527] LNet: Removed LNI 192.168.204.132@tcp [ 4831.630406] Key type .llcrypt unregistered [ 4831.632414] Key type ._llcrypt unregistered [ 4842.734290] Key type ._llcrypt registered [ 4842.735428] Key type .llcrypt registered [ 4842.775413] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 4843.246599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4843.256723] alg: No test for adler32 (adler32-zlib) [ 4844.149661] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 4844.255776] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 4845.856283] Key type lgssc registered [ 4846.482405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4850.398989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4850.411738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4851.645662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4853.544115] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4857.160970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4857.321560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4859.015628] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4861.635961] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4863.552475] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4866.930089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4867.129590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4869.974851] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4872.807326] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4881.382807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4881.386681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4881.390093] Lustre: Skipped 2 previous similar messages [ 4881.396050] Lustre: Skipped 2 previous similar messages [ 4883.202979] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.32@tcp (stopping) [ 4886.499650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4886.502961] Lustre: Skipped 2 previous similar messages [ 4886.732143] LustreError: 326008:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4886.770969] Lustre: server umount lustre-MDT0000 complete [ 4888.311773] LustreError: 325490:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4888.322204] LustreError: 325490:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4888.619125] LustreError: 323758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641753 with bad export cookie 15836378927951445196 [ 4888.626182] LustreError: 323758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4888.629249] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4891.617716] LustreError: 325490:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4891.618320] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4891.634102] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4893.431819] LustreError: 325490:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4894.780685] LustreError: 326209:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4894.782569] LustreError: 326209:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4894.858029] Lustre: server umount lustre-MDT0001 complete [ 4897.883576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4902.099674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4913.121169] LustreError: 323088:0:(client.c:1385:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bed4ee7f x1840923760681856/t0(0) o250->MGC192.168.204.132@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4913.298931] LustreError: 325376:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4913.309547] LustreError: 325376:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4913.347552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4913.396855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 4915.155307] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4918.498415] LustreError: 327018:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4918.507589] LustreError: 327018:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4919.316882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4919.522542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4921.169972] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4923.559860] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4923.627547] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 4929.652443] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4931.392687] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4934.390722] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 4939.237764] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4939.238548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4939.250814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4939.254288] Lustre: Skipped 3 previous similar messages [ 4942.262964] LustreError: 328501:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4942.266767] LustreError: 328501:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4942.365428] Lustre: server umount lustre-OST0000 complete [ 4944.354425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4944.361908] Lustre: Skipped 2 previous similar messages [ 4949.473979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4949.477697] Lustre: Skipped 3 previous similar messages [ 4950.288590] LustreError: 328703:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4950.292228] LustreError: 328703:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4950.327198] Lustre: server umount lustre-MDT0000 complete [ 4951.826966] LustreError: 324493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641816 with bad export cookie 15836378927951452364 [ 4951.833143] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4951.833630] LustreError: 324493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4952.022508] Lustre: server umount lustre-MDT0001 complete [ 4957.048321] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 4958.573068] Key type lgssc unregistered [ 4958.731612] LNet: 329432:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4958.735943] LNetError: 329432:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4958.745572] LNet: Removed LNI 192.168.204.132@tcp [ 4959.211419] Key type .llcrypt unregistered [ 4959.213277] Key type ._llcrypt unregistered [ 4971.489262] Key type ._llcrypt registered [ 4971.491082] Key type .llcrypt registered [ 4971.543967] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 4971.894099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4971.917518] alg: No test for adler32 (adler32-zlib) [ 4972.772025] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 4972.859138] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 4974.456167] Key type lgssc registered [ 4975.099880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4978.778128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4978.790835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4979.986322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4981.693575] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4985.908103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4986.097796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4987.775114] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 4990.514837] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4992.392499] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4994.530801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4994.532257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4994.543916] Lustre: Skipped 1 previous similar message [ 4995.554400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4995.558597] Lustre: Skipped 2 previous similar messages [ 5000.673916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5000.677914] Lustre: Skipped 1 previous similar message [ 5005.794087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5005.799088] Lustre: Skipped 1 previous similar message [ 5008.352109] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5008.402277] LustreError: 332178:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5008.437472] Lustre: server umount lustre-MDT0000 complete [ 5010.176907] LustreError: 332181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641875 with bad export cookie 14542172028747323240 [ 5010.182910] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5010.183777] LustreError: 332181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5010.273952] LustreError: 332383:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5010.276366] LustreError: 332383:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5010.356494] Lustre: server umount lustre-MDT0001 complete [ 5017.296788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5017.457365] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5017.477583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5017.598438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5019.181312] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5023.278292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5023.339092] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5023.359173] Lustre: 333154:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5023.508536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5025.336420] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5028.083791] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5029.933335] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5033.795361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5033.925631] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5033.932166] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5033.936778] LustreError: 334801:0:(tgt_mount.c:2419:server_fill_super()) Unable to start targets: -2 [ 5033.940729] LustreError: 334801:0:(tgt_mount.c:1913:server_put_super()) no obd lustre-OST0000 [ 5033.943995] LustreError: 334801:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 5034.027130] Lustre: server umount lustre-OST0000 complete [ 5034.029631] LustreError: 334801:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5038.314322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5038.443107] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5038.572530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5041.350272] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5043.960639] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5045.322774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 5047.822076] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5047.869379] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5047.911080] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5047.981637] Lustre: lustre-OST0001: new disk, initializing [ 5047.985530] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5048.044312] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5050.818421] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5052.970539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5052.977668] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5053.059906] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5053.415821] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5058.019176] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5058.023539] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5058.031477] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5058.535009] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5060.915192] LustreError: 336907:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5060.918949] LustreError: 336907:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5060.950991] Lustre: server umount lustre-OST0001 complete [ 5063.655228] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5063.657898] LustreError: 335268:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5063.662474] Lustre: Skipped 2 previous similar messages [ 5063.671687] LustreError: 335268:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 5068.775628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5068.779344] Lustre: Skipped 4 previous similar messages [ 5068.779455] LustreError: 335270:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5068.791127] LustreError: 335270:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5068.851194] LustreError: 337109:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5068.854974] LustreError: 337109:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5068.965069] Lustre: server umount lustre-OST0000 complete [ 5073.893887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5073.901273] Lustre: Skipped 1 previous similar message [ 5085.152196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5085.212813] LustreError: 337312:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5085.216468] LustreError: 337312:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5085.266973] Lustre: server umount lustre-MDT0000 complete [ 5087.127875] LustreError: 333144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755641952 with bad export cookie 14542172028747323779 [ 5087.134583] LustreError: 333144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5087.138307] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5087.355864] Lustre: server umount lustre-MDT0001 complete [ 5092.222308] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 5093.808594] Key type lgssc unregistered [ 5093.993608] LNet: 338043:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5093.998271] LNetError: 338043:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5094.006608] LNet: Removed LNI 192.168.204.132@tcp [ 5094.508059] Key type .llcrypt unregistered [ 5094.510188] Key type ._llcrypt unregistered [ 5108.324357] Key type ._llcrypt registered [ 5108.325811] Key type .llcrypt registered [ 5108.372860] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 5116.513288] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 5116.956560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5116.966709] alg: No test for adler32 (adler32-zlib) [ 5117.839261] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 5117.962571] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 5119.560145] Key type lgssc registered [ 5120.154510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5124.595427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5127.649622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5130.659745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5133.349783] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5136.679567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5136.713747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5136.724334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5137.857507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5137.876583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5137.934251] Lustre: lustre-MDT0000: new disk, initializing [ 5137.961742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5137.967532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5139.742853] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5145.113347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5145.147826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5145.202505] Lustre: 341953:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5145.257023] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5145.259175] Lustre: Skipped 1 previous similar message [ 5145.259508] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5145.296743] Lustre: lustre-MDT0001: new disk, initializing [ 5145.336143] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5145.351110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5145.354202] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5147.215713] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5151.255484] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5152.949125] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5155.557374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5155.598067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5155.767955] Lustre: lustre-OST0000: new disk, initializing [ 5155.771658] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5155.832894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5158.599320] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5162.660413] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5164.552416] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5165.076056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5165.080731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5165.153668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5165.739511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5167.658579] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5167.795306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5170.153642] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5170.155986] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5170.158251] LustreError: Skipped 1 previous similar message [ 5170.166798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5170.171630] Lustre: Skipped 1 previous similar message [ 5174.835161] LustreError: 344872:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5174.948391] Lustre: server umount lustre-OST0000 complete [ 5180.388933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5180.390283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5180.399854] Lustre: Skipped 2 previous similar messages [ 5185.506998] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5185.510565] Lustre: Skipped 2 previous similar messages [ 5190.627643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5190.633691] Lustre: Skipped 1 previous similar message [ 5191.136117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5191.196285] LustreError: 345073:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5191.200138] LustreError: 345073:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5191.244241] Lustre: server umount lustre-MDT0000 complete [ 5193.132392] LustreError: 341942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755642058 with bad export cookie 16675990698938061688 [ 5193.138969] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5193.143688] LustreError: 341942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5193.243535] LustreError: 345276:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5193.247126] LustreError: 345276:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5193.381939] Lustre: server umount lustre-MDT0001 complete [ 5197.654292] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 18:21:02 (1755642062) [ 5200.633967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5203.675078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5214.277357] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 5218.643232] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 5223.395058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5226.881241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5229.951960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5233.096112] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5236.576203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5236.621706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5236.770988] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5236.793383] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5236.858351] Lustre: lustre-MDT0000: new disk, initializing [ 5236.905700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5236.917965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5238.888434] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5244.706657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5244.750832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5244.802821] Lustre: 349034:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5244.848545] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5244.852127] Lustre: Skipped 1 previous similar message [ 5244.896288] Lustre: lustre-MDT0001: new disk, initializing [ 5244.957215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5244.977394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5244.985479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5246.838992] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5250.953076] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5252.668068] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5256.041683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5256.081062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5256.245455] Lustre: lustre-OST0000: new disk, initializing [ 5256.248395] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5256.297875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5259.154699] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5262.361495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5262.368715] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5262.424968] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5263.270641] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5265.242279] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5265.323498] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5267.119620] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5267.227540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5272.550136] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5272.550596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5272.554138] LustreError: Skipped 1 previous similar message [ 5272.554954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5272.565313] Lustre: Skipped 1 previous similar message [ 5272.573229] Lustre: Skipped 1 previous similar message [ 5274.159115] LustreError: 351944:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5274.162673] LustreError: 351944:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5274.275551] Lustre: server umount lustre-OST0000 complete [ 5277.668809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5277.677611] Lustre: Skipped 1 previous similar message [ 5282.190720] LustreError: 352146:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5282.197022] LustreError: 352146:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5282.255461] Lustre: server umount lustre-MDT0000 complete [ 5284.126169] LustreError: 349808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755642149 with bad export cookie 16675990698938062570 [ 5284.135665] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5284.137267] LustreError: 349808:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5284.342418] Lustre: server umount lustre-MDT0001 complete [ 5288.541796] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 18:22:33 (1755642153) [ 5292.181336] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 18:22:36 (1755642156) [ 5295.852474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5296.100148] LustreError: 353364:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5296.116478] LustreError: 353364:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 5296.137294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5297.875892] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5301.218075] LustreError: 353364:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5301.953609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5303.959118] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5306.529734] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5308.371494] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5311.622899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5314.445062] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5316.985947] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5326.191204] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5327.088714] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5327.935617] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5328.780485] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5329.710268] Lustre: Failing over lustre-MDT0000 [ 5329.872964] LustreError: 355783:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5329.877952] LustreError: 355783:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5329.914873] Lustre: server umount lustre-MDT0000 complete [ 5332.452656] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5332.454468] LustreError: 353363:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5332.464382] Lustre: Skipped 2 previous similar messages [ 5332.475510] LustreError: 353363:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5337.570991] LustreError: 353363:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5337.579936] LustreError: 353363:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 5342.690167] LustreError: 353363:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5342.698693] LustreError: 353363:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 5344.107916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5344.169719] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5344.314865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5344.321425] Lustre: Skipped 2 previous similar messages [ 5344.365486] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5346.087167] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5348.089479] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5349.361893] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 5349.372350] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5349.405937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5351.086172] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5351.835377] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5354.066048] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5355.030097] Lustre: Failing over lustre-MDT0000 [ 5355.282547] LustreError: 356858:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5355.284419] LustreError: 356858:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5355.314298] Lustre: server umount lustre-MDT0000 complete [ 5358.338923] LustreError: 353360:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5358.348266] LustreError: 353360:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5359.587171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5359.587489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5359.601192] Lustre: Skipped 1 previous similar message [ 5369.533096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5369.601461] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5369.796043] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5371.558105] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5373.690677] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5374.957486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 5374.963039] Lustre: Skipped 2 previous similar messages [ 5374.975416] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5375.008461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5376.738502] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5377.583935] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5385.190046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5385.190425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5385.204271] Lustre: Skipped 1 previous similar message [ 5385.208374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5385.211737] Lustre: Skipped 4 previous similar messages [ 5386.423675] Lustre: server umount lustre-MDT0000 complete [ 5388.205793] LustreError: 353345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755642253 with bad export cookie 16675990698938065237 [ 5388.210909] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5388.215916] LustreError: 353345:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5388.314680] LustreError: 358082:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5388.319990] LustreError: 358082:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5388.433294] Lustre: server umount lustre-MDT0001 complete [ 5400.678126] Lustre: server umount lustre-OST0000 complete [ 5408.276781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5408.525261] LustreError: 359111:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5408.535137] LustreError: 359111:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 8 previous similar messages [ 5408.566752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5408.570852] Lustre: Skipped 1 previous similar message [ 5410.493971] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5414.765842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5416.860320] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5419.623767] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5421.436315] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5424.777731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5427.625379] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5430.172383] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5433.196755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5443.553491] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5443.558106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5443.565183] Lustre: Skipped 2 previous similar messages [ 5443.568540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5448.475555] Lustre: server umount lustre-OST0000 complete [ 5456.397384] LustreError: 361563:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5456.401199] LustreError: 361563:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 5456.430916] Lustre: server umount lustre-MDT0000 complete [ 5458.244870] LustreError: 359090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755642323 with bad export cookie 16675990698938065944 [ 5458.252688] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5458.253397] LustreError: 359090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5458.442226] Lustre: server umount lustre-MDT0001 complete [ 5463.327345] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 5464.917989] Key type lgssc unregistered [ 5465.096811] LNet: 362291:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5465.101587] LNetError: 362291:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5465.115699] LNet: Removed LNI 192.168.204.132@tcp [ 5465.621911] Key type .llcrypt unregistered [ 5465.623965] Key type ._llcrypt unregistered [ 5478.588450] Key type ._llcrypt registered [ 5478.590534] Key type .llcrypt registered [ 5478.649670] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 5479.004301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5479.010460] alg: No test for adler32 (adler32-zlib) [ 5479.894827] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 5480.000862] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 5481.600216] Key type lgssc registered [ 5482.249767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5486.279158] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5486.292656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5487.553807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5489.450757] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5493.764451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5493.990454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5495.782695] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5498.333066] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5500.101238] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5503.429354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5503.628763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5506.348219] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5508.837137] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5512.885959] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5524.458818] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5524.461508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5524.466477] Lustre: Skipped 2 previous similar messages [ 5524.470146] Lustre: Skipped 1 previous similar message [ 5529.571569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5529.870107] LustreError: 366010:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5529.912815] Lustre: server umount lustre-MDT0000 complete [ 5531.710473] LustreError: 364471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755642396 with bad export cookie 18338812689348617293 [ 5531.717102] LustreError: 364471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5531.718651] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5531.814393] LustreError: 366212:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5531.818226] LustreError: 366212:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5531.929438] Lustre: server umount lustre-MDT0001 complete [ 5544.053619] LustreError: 366412:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5544.057299] LustreError: 366412:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5544.138874] Lustre: server umount lustre-OST0000 complete [ 5549.860072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5550.070399] LustreError: 367141:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5550.075996] LustreError: 367141:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5550.105399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5551.654918] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5555.170990] LustreError: 367141:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5555.229620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5555.388116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5556.967312] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5559.244187] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5560.891472] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5564.226817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5564.421948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5567.030690] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5569.666196] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5574.707248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5584.870201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5584.874238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5584.877554] Lustre: Skipped 2 previous similar messages [ 5584.883384] Lustre: Skipped 4 previous similar messages [ 5588.812558] LustreError: 369471:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5588.816414] LustreError: 369471:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5588.853503] Lustre: server umount lustre-MDT0000 complete [ 5589.986378] LustreError: 367140:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5589.995728] LustreError: 367140:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5590.689347] LustreError: 367122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755642455 with bad export cookie 18338812689348618406 [ 5590.696261] LustreError: 367122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5590.699408] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5590.898166] Lustre: server umount lustre-MDT0001 complete [ 5603.063503] LustreError: 369877:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5603.067425] LustreError: 369877:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5603.180326] Lustre: server umount lustre-OST0000 complete [ 5610.855189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5611.113716] LustreError: 370706:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5611.157884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5613.015910] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5616.610618] LustreError: 370706:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5617.352728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5619.497260] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5622.165941] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5623.999895] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5627.331709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5630.300951] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5632.980352] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5636.785611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5642.216261] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5642.218052] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5642.221821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5642.227943] Lustre: Skipped 1 previous similar message [ 5645.938169] LustreError: 372934:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5645.942072] LustreError: 372934:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5646.048124] Lustre: server umount lustre-OST0000 complete [ 5648.353502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5648.358206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5648.366376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5648.370018] Lustre: Skipped 1 previous similar message [ 5652.452846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5657.570485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5657.574178] Lustre: Skipped 3 previous similar messages [ 5662.176131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5662.231055] LustreError: 373135:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5662.234833] LustreError: 373135:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5662.275277] Lustre: server umount lustre-MDT0000 complete [ 5662.690933] LustreError: 370706:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5662.699615] LustreError: 370706:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5663.915207] LustreError: 370686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755642528 with bad export cookie 18338812689348619414 [ 5663.917716] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5663.921677] LustreError: 370686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5664.070354] Lustre: server umount lustre-MDT0001 complete [ 5668.976798] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 5670.665648] Key type lgssc unregistered [ 5670.845545] LNet: 373865:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5670.850277] LNetError: 373865:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5670.860810] LNet: Removed LNI 192.168.204.132@tcp [ 5671.370630] Key type .llcrypt unregistered [ 5671.372760] Key type ._llcrypt unregistered [ 5684.129305] Key type ._llcrypt registered [ 5684.130816] Key type .llcrypt registered [ 5684.197207] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 5684.595825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5684.606539] alg: No test for adler32 (adler32-zlib) [ 5685.473129] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 5685.594450] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 5687.184224] Key type lgssc registered [ 5687.761251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5691.807717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5691.819183] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5691.824677] LustreError: 375257:0:(osd_handler.c:8374:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5691.836880] LustreError: 375257:0:(obd_config.c:779:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5691.841167] LustreError: 375257:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5691.845608] LustreError: 375257:0:(tgt_mount.c:2387:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5691.851171] LustreError: 375257:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5695.485655] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5695.489424] LustreError: 375654:0:(osd_handler.c:8374:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5695.499454] LustreError: 375654:0:(obd_config.c:779:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5695.503050] LustreError: 375654:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5695.506793] LustreError: 375654:0:(tgt_mount.c:2387:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5695.510595] LustreError: 375654:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5704.018246] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 5705.663159] Key type lgssc unregistered [ 5705.839903] LNet: 376634:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5705.846845] LNetError: 376634:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5705.859656] LNet: Removed LNI 192.168.204.132@tcp [ 5706.361624] Key type .llcrypt unregistered [ 5706.363749] Key type ._llcrypt unregistered [ 5714.849286] Key type ._llcrypt registered [ 5714.851397] Key type .llcrypt registered [ 5714.929279] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 5723.054866] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 5723.670345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5723.680276] alg: No test for adler32 (adler32-zlib) [ 5724.580255] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 5724.705324] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 5726.320192] Key type lgssc registered [ 5726.942718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5731.660977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5735.094467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5738.249361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5741.567583] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5745.043287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5745.077694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5745.088418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5746.227227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5746.247943] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5746.304167] Lustre: lustre-MDT0000: new disk, initializing [ 5746.346991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5746.358890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5748.199338] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5753.854573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5753.887511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5753.943571] Lustre: 379727:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5753.989506] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5753.992960] Lustre: Skipped 1 previous similar message [ 5754.036899] Lustre: lustre-MDT0001: new disk, initializing [ 5754.084963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5754.108526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5754.114606] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5755.979925] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5760.053974] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5761.828453] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5765.133472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5765.177770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5765.337311] Lustre: lustre-OST0000: new disk, initializing [ 5765.341072] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5765.397277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5768.050080] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5769.173316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5769.179444] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5769.245874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5772.136636] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5774.114585] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5774.253034] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5776.220215] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5776.357641] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5779.429227] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5779.431890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5779.433769] LustreError: Skipped 1 previous similar message [ 5779.441905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5779.447038] Lustre: Skipped 1 previous similar message [ 5783.412167] LustreError: 382636:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5783.519693] Lustre: server umount lustre-OST0000 complete [ 5789.668522] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5789.670343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5789.675704] Lustre: Skipped 2 previous similar messages [ 5791.444925] LustreError: 382838:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5791.448611] LustreError: 382838:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5791.494834] Lustre: server umount lustre-MDT0000 complete [ 5793.367372] LustreError: 379717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755642658 with bad export cookie 10366105440775547617 [ 5793.374209] LustreError: 379717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5793.375309] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5793.471346] LustreError: 383040:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5793.474567] LustreError: 383040:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5793.595060] Lustre: server umount lustre-MDT0001 complete [ 5797.495867] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5798.338936] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 18:31:03 (1755642663) [ 5801.618651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5801.924124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5803.842528] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5807.075500] LustreError: 383760:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5807.084246] LustreError: 383760:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5807.998616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5808.197279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5810.010433] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5812.632965] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5814.367763] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5817.714688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5817.919750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5820.787109] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5823.391123] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5826.801332] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5826.844117] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5826.951865] Lustre: lustre-OST0001: new disk, initializing [ 5826.955375] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5829.781269] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5832.247079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5832.252450] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5832.310469] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5833.778167] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5837.282661] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5837.287051] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5837.294724] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5837.298431] Lustre: Skipped 1 previous similar message [ 5838.068191] LustreError: 386934:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5838.073421] LustreError: 386934:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5838.105301] Lustre: server umount lustre-OST0001 complete [ 5839.081766] LustreError: 386751:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.32@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5842.925949] LustreError: 385484:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5842.938992] LustreError: 385484:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5848.044224] LustreError: 386751:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5848.053373] LustreError: 386751:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5852.129592] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5852.134187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5852.141342] Lustre: Skipped 1 previous similar message [ 5852.144812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5852.148306] Lustre: Skipped 1 previous similar message [ 5853.163289] LustreError: 385484:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5853.172802] LustreError: 385484:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5857.010964] LustreError: 387158:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5857.014685] LustreError: 387158:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5857.128178] Lustre: server umount lustre-OST0000 complete [ 5859.297334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5859.301983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5859.309139] Lustre: Skipped 1 previous similar message [ 5859.312648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5859.316263] Lustre: Skipped 2 previous similar messages [ 5865.232108] LustreError: 387359:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5865.236039] LustreError: 387359:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5865.288639] Lustre: server umount lustre-MDT0000 complete [ 5867.167755] LustreError: 383741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755642732 with bad export cookie 10366105440775548499 [ 5867.179038] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5873.487339] Lustre: server umount lustre-MDT0001 complete [ 5878.381243] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 5880.085301] Key type lgssc unregistered [ 5880.282622] LNet: 388088:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5880.287328] LNetError: 388088:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5880.302730] LNet: Removed LNI 192.168.204.132@tcp [ 5880.804501] Key type .llcrypt unregistered [ 5880.806609] Key type ._llcrypt unregistered [ 5897.018345] Key type ._llcrypt registered [ 5897.020302] Key type .llcrypt registered [ 5897.080876] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 5905.542816] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 5906.116539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5906.126419] alg: No test for adler32 (adler32-zlib) [ 5907.012616] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 5907.123638] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 5908.712163] Key type lgssc registered [ 5909.262891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5914.136646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5917.632185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5920.726899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5924.018911] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5927.507723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5927.544126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5927.552466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5928.674089] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5928.692487] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5928.731924] Lustre: lustre-MDT0000: new disk, initializing [ 5928.773038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5928.784226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5930.685905] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5936.462179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5936.510073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5936.569602] Lustre: 391999:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5936.619782] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5936.623332] Lustre: Skipped 1 previous similar message [ 5936.674191] Lustre: lustre-MDT0001: new disk, initializing [ 5936.731233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5936.751310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5936.756596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5938.637463] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5942.897234] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5944.812317] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5948.244859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5948.288341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5948.464340] Lustre: lustre-OST0000: new disk, initializing [ 5948.468401] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5948.525976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5951.331547] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5954.571200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5954.576628] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5954.634912] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5955.421068] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5957.399201] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5957.536520] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5959.403650] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5959.540639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5964.777205] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5964.777554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5964.781458] LustreError: Skipped 1 previous similar message [ 5964.783063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5964.788348] Lustre: Skipped 1 previous similar message [ 5966.514785] LustreError: 394910:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5966.626958] Lustre: server umount lustre-OST0000 complete [ 5969.892993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5969.894643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5969.900047] Lustre: Skipped 1 previous similar message [ 5969.905879] Lustre: Skipped 1 previous similar message [ 5974.605871] LustreError: 395113:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5974.609565] LustreError: 395113:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5974.649613] Lustre: server umount lustre-MDT0000 complete [ 5976.421147] LustreError: 393734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755642841 with bad export cookie 5514398066248590319 [ 5976.427190] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5976.521810] LustreError: 395315:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5976.526700] LustreError: 395315:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5976.636396] Lustre: server umount lustre-MDT0001 complete [ 5980.531708] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 18:34:05 (1755642845) [ 5985.036066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5989.100442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5989.351252] Lustre: lustre-MDT0000: new disk, initializing [ 5989.366962] LustreError: 396581:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5989.376798] LustreError: 396581:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 5989.413674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5991.318455] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 5994.467722] LustreError: 396581:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5995.586975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5995.769154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5997.692969] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6000.282368] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6001.907810] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6005.220784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6005.221884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6005.227792] Lustre: Skipped 1 previous similar message [ 6009.037206] LustreError: 397912:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6009.041241] LustreError: 397912:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6009.079259] Lustre: server umount lustre-MDT0000 complete [ 6010.884359] LustreError: 396562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755642875 with bad export cookie 5514398066248591187 [ 6010.890974] LustreError: 396562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6010.891571] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6011.123790] Lustre: server umount lustre-MDT0001 complete [ 6015.571320] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 18:34:40 (1755642880) [ 6018.893832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6019.144244] LustreError: 398836:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6019.189441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6021.061777] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6024.677504] LustreError: 398836:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6025.289086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6027.324720] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6029.862968] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6031.595955] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6034.853891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6037.706781] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6040.266179] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6049.190079] Lustre: Permanently deactivating lustre-OST0000 [ 6049.199099] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6055.431654] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6057.289534] LustreError: 401510:0:(mgs_llog.c:1937:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6057.296325] LustreError: 401510:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 6057.301754] LustreError: 401510:0:(mgs_handler.c:1161:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6058.155984] LustreError: 401558:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 6058.160646] LustreError: 401558:0:(mgs_handler.c:1161:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6065.143233] LustreError: 401655:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6065.148689] LustreError: 401655:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6065.255502] Lustre: server umount lustre-OST0000 complete [ 6066.581903] LustreError: 401808:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 6066.586219] LustreError: 401808:0:(mgs_handler.c:1161:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6070.244432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6070.245596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6070.251305] Lustre: Skipped 1 previous similar message [ 6070.257017] Lustre: Skipped 4 previous similar messages [ 6075.362730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6075.366366] Lustre: Skipped 1 previous similar message [ 6077.414904] LustreError: 401905:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6077.418489] LustreError: 401905:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6077.464295] Lustre: server umount lustre-MDT0000 complete [ 6079.360674] LustreError: 399550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755642944 with bad export cookie 5514398066248591726 [ 6079.369049] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6079.372105] LustreError: 399550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6089.814986] Lustre: server umount lustre-MDT0001 complete [ 6094.145342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6096.183032] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6097.562221] LustreError: 402945:0:(mgs_handler.c:1161:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6100.726812] Lustre: 403283:0:(mgs_llog.c:1698:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.132@tcp [ 6101.757540] LustreError: 403380:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6101.761307] LustreError: 403380:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6101.798747] Lustre: server umount MGS complete [ 6107.634331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6107.895877] LustreError: 404008:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6107.905555] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6107.909969] Lustre: Skipped 1 previous similar message [ 6107.948502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6107.952397] Lustre: Skipped 2 previous similar messages [ 6109.785661] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6114.024693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6114.185333] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6115.955710] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6118.633933] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6120.441968] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6121.415784] Lustre: Permanently reactivating lustre-OST0000 [ 6121.425763] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6121.429620] Lustre: Skipped 2 previous similar messages [ 6150.112247] LustreError: 405648:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6153.686757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6153.902947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6153.907336] Lustre: Skipped 1 previous similar message [ 6156.699030] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6159.328243] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6174.689546] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6174.694127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6174.702203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6175.717619] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6177.712198] LustreError: 407375:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6177.716246] LustreError: 407375:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6177.823819] Lustre: server umount lustre-OST0000 complete [ 6180.838070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6185.854262] Lustre: server umount lustre-MDT0000 complete [ 6185.954082] LustreError: 404007:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6185.961569] LustreError: 404007:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6187.684249] LustreError: 403990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755643052 with bad export cookie 5514398066248592685 [ 6187.690835] LustreError: 403990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6187.693828] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6187.911471] Lustre: server umount lustre-MDT0001 complete [ 6192.818776] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 6194.561193] Key type lgssc unregistered [ 6194.780727] LNet: 408306:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6194.785293] LNetError: 408306:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6194.796762] LNet: Removed LNI 192.168.204.132@tcp [ 6195.326260] Key type .llcrypt unregistered [ 6195.328424] Key type ._llcrypt unregistered [ 6204.139774] Key type ._llcrypt registered [ 6204.142475] Key type .llcrypt registered [ 6204.204730] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 6212.561411] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 6212.913056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6212.959728] alg: No test for adler32 (adler32-zlib) [ 6213.852517] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 6213.966492] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 6215.544247] Key type lgssc registered [ 6216.182589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6221.070799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6224.604886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6227.804709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6231.144645] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6234.600124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6234.642842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6234.654107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6235.792409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6235.813381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6235.867300] Lustre: lustre-MDT0000: new disk, initializing [ 6235.912480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6235.922560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6237.852668] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6243.541270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6243.586581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6243.641829] Lustre: 411401:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6243.693321] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6243.697484] Lustre: Skipped 1 previous similar message [ 6243.750567] Lustre: lustre-MDT0001: new disk, initializing [ 6243.808767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6243.831920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6243.838264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6245.677410] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6249.851619] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6251.654838] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6255.042809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6255.082040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6255.248910] Lustre: lustre-OST0000: new disk, initializing [ 6255.252608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6255.303691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6258.038876] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6261.263933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6261.270151] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6261.330389] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6262.238573] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6264.256347] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6264.372635] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6266.315742] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6266.461460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6271.462425] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6271.463573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6271.473257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6271.476341] Lustre: Skipped 1 previous similar message [ 6273.523241] LustreError: 414307:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6273.628824] Lustre: server umount lustre-OST0000 complete [ 6276.580373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6276.582317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6276.587643] Lustre: Skipped 2 previous similar messages [ 6281.617461] LustreError: 414510:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6281.621203] LustreError: 414510:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6281.669447] Lustre: server umount lustre-MDT0000 complete [ 6283.504154] LustreError: 412175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755643148 with bad export cookie 18010918861862677005 [ 6283.511053] LustreError: 412175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6283.513919] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6283.621814] LustreError: 414712:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6283.624715] LustreError: 414712:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6283.740165] Lustre: server umount lustre-MDT0001 complete [ 6287.880632] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6288.715240] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 18:39:13 (1755643153) [ 6291.975575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6292.223411] LustreError: 415431:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6292.235678] LustreError: 415431:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 6292.263953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6294.062941] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6297.572456] LustreError: 415431:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6298.220946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6298.411557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6300.268553] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6302.833085] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6304.585905] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6307.980694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6308.194415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6311.028202] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6313.532768] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6314.478433] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6320.499793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6322.664320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6322.664753] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6322.665954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6322.665963] Lustre: Skipped 1 previous similar message [ 6322.671535] Lustre: Skipped 1 previous similar message [ 6322.675734] LustreError: Skipped 1 previous similar message [ 6327.779611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6327.783266] Lustre: Skipped 1 previous similar message [ 6328.368185] LustreError: 417851:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6328.371920] LustreError: 417851:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6328.474646] Lustre: server umount lustre-OST0000 complete [ 6332.899665] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6332.908418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6332.912024] Lustre: Skipped 1 previous similar message [ 6336.457399] LustreError: 418053:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6336.460335] LustreError: 418053:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6336.495725] Lustre: server umount lustre-MDT0000 complete [ 6338.018277] LustreError: 415430:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6338.026928] LustreError: 415430:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6338.316105] LustreError: 417036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755643203 with bad export cookie 18010918861862677887 [ 6338.322531] LustreError: 417036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6338.322854] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6338.551777] Lustre: server umount lustre-MDT0001 complete [ 6343.622697] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 6345.380536] Key type lgssc unregistered [ 6345.577773] LNet: 418783:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6345.582481] LNetError: 418783:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6345.592850] LNet: Removed LNI 192.168.204.132@tcp [ 6346.108587] Key type .llcrypt unregistered [ 6346.110563] Key type ._llcrypt unregistered [ 6356.167348] Key type ._llcrypt registered [ 6356.169438] Key type .llcrypt registered [ 6356.237228] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 6356.423369] Key type .llcrypt unregistered [ 6356.425450] Key type ._llcrypt unregistered [ 6365.374668] Key type ._llcrypt registered [ 6365.376560] Key type .llcrypt registered [ 6365.450348] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 6365.832572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6365.877068] alg: No test for adler32 (adler32-zlib) [ 6366.778206] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 6366.901389] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 6368.512271] Key type lgssc registered [ 6369.159395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6373.224196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6373.236801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6374.499413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6376.409343] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6380.631437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6380.832836] LustreError: 421548:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6380.839319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6380.842168] LustreError: 421548:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6383.561487] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6384.930393] LustreError: 420937:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6384.937689] LustreError: 420937:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6384.940389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6386.142881] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6389.410664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6389.642271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6391.575263] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6399.973951] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6399.979730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6399.982509] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6399.989076] Lustre: Skipped 1 previous similar message [ 6405.094483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6405.098151] Lustre: Skipped 1 previous similar message [ 6405.492220] LustreError: 422957:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6405.596361] Lustre: server umount lustre-OST0000 complete [ 6410.213199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6410.213409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6410.215511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6410.216820] Lustre: Skipped 2 previous similar messages [ 6413.585995] LustreError: 423158:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6413.589953] LustreError: 423158:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6413.631090] Lustre: server umount lustre-MDT0000 complete [ 6415.331313] LustreError: 420919:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6415.340461] LustreError: 420919:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6415.464843] LustreError: 420900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755643280 with bad export cookie 1934046958524955811 [ 6415.471688] LustreError: 420900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6415.475532] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6415.560722] LustreError: 423360:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6415.562982] LustreError: 423360:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6415.688758] Lustre: server umount lustre-MDT0001 complete [ 6420.602147] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 6422.236929] Key type lgssc unregistered [ 6422.426678] LNet: 423886:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6422.430970] LNetError: 423886:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6422.442875] LNet: Removed LNI 192.168.204.132@tcp [ 6422.927470] Key type .llcrypt unregistered [ 6422.929515] Key type ._llcrypt unregistered [ 6434.944879] Key type ._llcrypt registered [ 6434.946557] Key type .llcrypt registered [ 6435.002243] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 6435.458825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6435.530348] alg: No test for adler32 (adler32-zlib) [ 6436.413388] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 6436.523329] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 6438.128176] Key type lgssc registered [ 6438.759883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6442.598344] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6442.609913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6459.104417] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6465.248211] LustreError: 425228:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.132@tcp: failed processing log, type 4: rc = -110 [ 6495.968261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6498.690151] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6501.356299] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6504.730979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6504.999580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6505.047140] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6506.904488] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6510.051319] LustreError: 425255:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6510.060149] LustreError: 425255:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6511.073197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6511.078361] LustreError: 426146:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6511.090677] LustreError: 426146:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6511.299938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6513.132731] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6515.685916] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6517.488380] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6521.832730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6521.835187] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6521.836674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6521.842020] Lustre: Skipped 1 previous similar message [ 6521.846419] Lustre: Skipped 1 previous similar message [ 6526.450191] LustreError: 427490:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 6526.555137] Lustre: server umount lustre-OST0000 complete [ 6531.043799] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6531.048235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6531.055925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6532.067199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6532.068769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6534.545090] LustreError: 427693:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6534.548883] LustreError: 427693:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6534.594965] Lustre: server umount lustre-MDT0000 complete [ 6536.470183] LustreError: 426851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755643401 with bad export cookie 15801983323207071186 [ 6536.478941] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6536.564862] LustreError: 427896:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6536.568500] LustreError: 427896:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6536.692896] Lustre: server umount lustre-MDT0001 complete [ 6541.911234] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 6543.639800] Key type lgssc unregistered [ 6543.819706] LNet: 428422:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6543.824417] LNetError: 428422:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6543.834774] LNet: Removed LNI 192.168.204.132@tcp [ 6544.362082] Key type .llcrypt unregistered [ 6544.364058] Key type ._llcrypt unregistered [ 6556.460893] Key type ._llcrypt registered [ 6556.462965] Key type .llcrypt registered [ 6556.532553] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 6557.136816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6557.144286] alg: No test for adler32 (adler32-zlib) [ 6558.007303] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 6558.107596] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 6559.712170] Key type lgssc registered [ 6560.380794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6564.380507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6564.393082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6565.635209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6567.597170] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6571.885338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6572.106596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6574.071317] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6576.752407] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6578.526898] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6581.869428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6582.068854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6584.755487] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6587.309935] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6590.257921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6595.557608] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6595.561143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6595.564579] Lustre: Skipped 2 previous similar messages [ 6595.568027] Lustre: Skipped 1 previous similar message [ 6596.579104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6596.582785] Lustre: Skipped 2 previous similar messages [ 6598.916761] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.32@tcp (stopping) [ 6600.911651] LustreError: 432018:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6600.959768] Lustre: server umount lustre-MDT0000 complete [ 6601.698939] LustreError: 429791:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6601.708331] LustreError: 429791:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 6602.074565] LustreError: 429772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.32@tcp arrived at 1755643466 with bad export cookie 10558193796782504704 [ 6602.082112] LustreError: 429772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6603.039843] LustreError: 429772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755643467 with bad export cookie 10558193796782504046 [ 6603.049073] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6606.818443] LustreError: 429791:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6606.819350] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6606.827157] LustreError: 429791:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6606.834889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6608.866217] LustreError: 431241:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6608.875266] LustreError: 431241:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6609.139605] LustreError: 432221:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6609.143324] LustreError: 432221:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6609.247466] Lustre: server umount lustre-OST0000 complete [ 6612.920306] LustreError: 430500:0:(client.c:1375:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000002314b27 x1840925557943040/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6612.937223] LustreError: 432574:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6612.940920] LustreError: 432574:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6613.075718] Lustre: server umount lustre-MDT0001 complete [ 6617.906883] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 6619.531896] Key type lgssc unregistered [ 6619.724762] LNet: 433100:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6619.729489] LNetError: 433100:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6619.742716] LNet: Removed LNI 192.168.204.132@tcp [ 6620.251665] Key type .llcrypt unregistered [ 6620.253651] Key type ._llcrypt unregistered [ 6632.381370] Key type ._llcrypt registered [ 6632.383305] Key type .llcrypt registered [ 6632.450334] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 6632.949321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6632.978317] alg: No test for adler32 (adler32-zlib) [ 6633.876286] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 6634.011871] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 6635.608270] Key type lgssc registered [ 6636.272558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6640.297473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6640.312402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6641.579872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6643.486751] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6647.834048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6648.062952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6650.018141] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6652.552209] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6654.236044] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6657.646671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6657.850321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6660.637555] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6663.250350] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6667.058635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6671.841878] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6671.846284] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6671.854219] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6672.356339] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6672.359149] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6674.680653] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.32@tcp (stopping) [ 6674.685404] Lustre: Skipped 1 previous similar message [ 6676.687681] LustreError: 436697:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6676.723301] Lustre: server umount lustre-MDT0001 complete [ 6677.476298] LustreError: 434469:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6677.485441] LustreError: 434469:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 6682.595189] LustreError: 434469:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6682.596076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6682.598820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6682.600628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6685.178151] LustreError: 436903:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6685.182060] LustreError: 436903:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6685.292391] Lustre: server umount lustre-OST0000 complete [ 6687.248524] LustreError: 437104:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6687.251810] LustreError: 437104:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6687.387456] Lustre: server umount lustre-MDT0000 complete [ 6693.992203] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 6695.647924] Key type lgssc unregistered [ 6695.827951] LNet: 437782:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6695.832509] LNetError: 437782:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6695.842947] LNet: Removed LNI 192.168.204.132@tcp [ 6696.350199] Key type .llcrypt unregistered [ 6696.352336] Key type ._llcrypt unregistered [ 6707.885555] Key type ._llcrypt registered [ 6707.887614] Key type .llcrypt registered [ 6707.951790] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 6716.053519] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 6716.406028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6716.417412] alg: No test for adler32 (adler32-zlib) [ 6717.326839] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 6717.443549] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 6719.048184] Key type lgssc registered [ 6719.662186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6724.311922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6727.687532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6730.756045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6734.016793] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6738.860274] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 6743.427986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6746.767913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6746.803365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6746.818789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6747.929435] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6747.944088] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6747.993677] Lustre: lustre-MDT0000: new disk, initializing [ 6748.028709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6748.038131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6749.881266] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6754.590347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6757.863130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6757.913089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6758.049365] Lustre: lustre-OST0000: new disk, initializing [ 6758.051795] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6758.053977] Lustre: Skipped 1 previous similar message [ 6758.092429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6760.152173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6760.156088] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6760.169668] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6760.787882] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6764.903842] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6769.332997] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6772.570966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6772.615095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6772.671667] Lustre: 441709:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6772.730873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6772.790583] Lustre: lustre-MDT0001: new disk, initializing [ 6772.875552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6772.902066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6772.908065] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6774.712301] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6779.494306] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6779.640680] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6781.617597] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6781.747659] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6786.533909] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6786.534510] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6786.536182] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6786.541854] Lustre: Skipped 1 previous similar message [ 6788.871841] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.32@tcp (stopping) [ 6788.875776] Lustre: Skipped 1 previous similar message [ 6791.653494] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6791.655280] Lustre: Skipped 1 previous similar message [ 6793.975540] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.32@tcp (stopping) [ 6798.304187] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6798.355649] LustreError: 445101:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6798.391740] Lustre: server umount lustre-MDT0001 complete [ 6807.015342] LustreError: 441716:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6807.022735] LustreError: 441716:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 6811.617390] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6811.621813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6811.629483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6811.632835] Lustre: Skipped 2 previous similar messages [ 6812.133247] LustreError: 441721:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6817.253320] LustreError: 441721:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6817.715133] LustreError: 445375:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 6817.718809] LustreError: 445375:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6817.834943] Lustre: server umount lustre-OST0000 complete [ 6825.998111] LustreError: 445576:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6826.002023] LustreError: 445576:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6826.124429] Lustre: server umount lustre-MDT0000 complete [ 6832.221600] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 6833.609604] Key type lgssc unregistered [ 6833.768581] LNet: 446256:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6833.773449] LNetError: 446256:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6833.782566] LNet: Removed LNI 192.168.204.132@tcp [ 6834.209166] Key type .llcrypt unregistered [ 6834.210620] Key type ._llcrypt unregistered [ 6851.217403] Key type ._llcrypt registered [ 6851.219298] Key type .llcrypt registered [ 6851.273551] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 6851.429291] Key type .llcrypt unregistered [ 6851.430918] Key type ._llcrypt unregistered [ 6859.464633] Key type ._llcrypt registered [ 6859.466140] Key type .llcrypt registered [ 6859.519175] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 6859.896309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6859.925663] alg: No test for adler32 (adler32-zlib) [ 6860.777662] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 6860.880787] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 6862.496180] Key type lgssc registered [ 6863.133516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6867.131264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6870.178719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6873.094533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6876.395727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6876.434376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6876.444710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6877.591393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6877.612995] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6877.675219] Lustre: lustre-MDT0000: new disk, initializing [ 6877.716949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6877.729493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6879.652859] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6885.259445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6885.301657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6885.361622] Lustre: 449562:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6885.404652] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6885.407957] Lustre: Skipped 1 previous similar message [ 6885.464152] Lustre: lustre-MDT0001: new disk, initializing [ 6885.512415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6885.533754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6885.540070] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6887.350278] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6891.323830] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6892.900840] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6896.185697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6896.230926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6896.388551] Lustre: lustre-OST0000: new disk, initializing [ 6896.392168] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6896.441714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6899.237690] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6903.303323] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6904.340716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6904.349288] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6904.417858] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6911.457292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6911.461530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6911.469196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6914.534780] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6914.538514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6914.541779] Lustre: Skipped 1 previous similar message [ 6914.546612] Lustre: Skipped 1 previous similar message [ 6917.390589] LustreError: 451990:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6917.437934] Lustre: server umount lustre-MDT0000 complete [ 6919.280625] LustreError: 449554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755643784 with bad export cookie 2824734640481119997 [ 6919.286906] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6919.288174] LustreError: 449554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6919.386714] LustreError: 452191:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6919.390817] LustreError: 452191:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6919.516573] Lustre: server umount lustre-MDT0001 complete [ 6931.575545] LustreError: 452392:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6931.579167] LustreError: 452392:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6931.692385] Lustre: server umount lustre-OST0000 complete [ 6936.570680] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 18:50:01 (1755643801) [ 6936.960096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6940.258690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6940.560778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6942.263931] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6945.762740] LustreError: 453221:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6945.771942] LustreError: 453221:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6946.436200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6946.643967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6948.522493] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6951.100514] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6952.840649] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6956.100684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6956.145140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6956.262942] Lustre: Found index 0 for lustre-OST0000, updating log [ 6956.362682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6958.866495] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 6961.450148] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6965.613993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6970.853717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6970.855985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6970.856089] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6970.863759] Lustre: Skipped 1 previous similar message [ 6974.579146] LustreError: 455463:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6974.582829] LustreError: 455463:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6974.682460] Lustre: server umount lustre-OST0000 complete [ 6975.985093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6976.994283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6976.995396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6977.004206] Lustre: Skipped 1 previous similar message [ 6977.010608] Lustre: Skipped 4 previous similar messages [ 6983.137973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6983.143230] Lustre: Skipped 3 previous similar messages [ 6991.328192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6991.379453] LustreError: 455719:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6991.383281] LustreError: 455719:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6991.415925] Lustre: server umount lustre-MDT0000 complete [ 6993.193107] LustreError: 455721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755643858 with bad export cookie 2824734640481121376 [ 6993.198551] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6993.199658] LustreError: 455721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6993.404293] Lustre: server umount lustre-MDT0001 complete [ 6997.432486] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 18:51:02 (1755643862) [ 6997.909639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6998.390710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6998.868754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6999.340413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6999.810700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7000.319417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7000.799916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7001.283845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7001.768197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7002.248151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7002.739524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7003.215591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7003.678567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7004.154276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7004.642766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7005.105774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7005.605060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7006.079677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7006.574081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7007.056704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7007.549901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7008.041063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7008.522734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7008.987302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7009.465644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7009.943319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7010.421050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7010.906172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7011.383664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7011.854981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7012.338565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7012.828206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7013.269634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7013.656721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7014.132131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7014.625609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7015.092782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7015.562474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7016.041110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7016.531620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7017.013611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7017.472781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7017.960421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7018.440370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7018.920260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7019.401040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7019.864868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7020.329963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7020.785610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7021.254977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7021.742592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7025.404606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7025.656069] LustreError: 459311:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7025.695250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7027.385978] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7030.753895] LustreError: 459311:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7031.457453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7033.315752] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7035.857853] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7037.548308] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7040.793687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7040.845696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7040.965007] Lustre: Found index 0 for lustre-OST0000, updating log [ 7043.795700] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7046.368770] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7048.678783] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7048.680120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7048.691382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7048.694752] Lustre: Skipped 2 previous similar messages [ 7049.391145] LustreError: 461599:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7049.393056] LustreError: 461599:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7049.490399] Lustre: server umount lustre-OST0000 complete [ 7052.259275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7057.451727] Lustre: server umount lustre-MDT0000 complete [ 7058.402093] LustreError: 459311:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7058.410538] LustreError: 459311:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 7059.250240] LustreError: 459290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755643924 with bad export cookie 2824734640481122447 [ 7059.255502] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7059.256670] LustreError: 459290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7059.450507] Lustre: server umount lustre-MDT0001 complete [ 7060.772457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7068.168547] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 7072.865661] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 7077.697484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7081.134228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7084.356576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7087.656602] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7091.053748] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 18:52:35 (1755643955) [ 7099.166377] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 7100.799607] Key type lgssc unregistered [ 7103.008915] LNet: 465557:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7103.013638] LNetError: 465557:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7226.922230] LNet: Removed LNI 192.168.204.132@tcp [ 7227.393174] Key type .llcrypt unregistered [ 7227.394772] Key type ._llcrypt unregistered [ 7235.547745] Key type ._llcrypt registered [ 7235.549604] Key type .llcrypt registered [ 7235.615396] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 7236.009034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7236.049470] alg: No test for adler32 (adler32-zlib) [ 7236.936865] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 7237.031914] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 7238.608182] Key type lgssc registered [ 7239.228202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7241.473222] LNet: Added LNI 192.168.204.132@tcp100 [8/256/0/180] [ 7242.278203] Lustre: DEBUG MARKER: SKIP: conf-sanity test_73c unable to configure net [ 7250.337331] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 7251.926581] Key type lgssc unregistered [ 7252.154606] LNet: 467410:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7252.158918] LNetError: 467410:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7252.163083] LNet: Removed LNI 192.168.204.132@tcp [ 7252.172645] LNet: Removed LNI 192.168.204.132@tcp100 [ 7252.613729] Key type .llcrypt unregistered [ 7252.615697] Key type ._llcrypt unregistered [ 7262.509909] Key type ._llcrypt registered [ 7262.511044] Key type .llcrypt registered [ 7262.556261] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 7269.552840] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 7269.867572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7269.912107] alg: No test for adler32 (adler32-zlib) [ 7270.778813] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 7270.865038] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 7272.440243] Key type lgssc registered [ 7273.006814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7277.397572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7280.529720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7283.446886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7286.554442] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7289.601538] Lustre: DEBUG MARKER: == conf-sanity test 74: Test per-device adaptive timeout parameters ========================================================== 18:55:54 (1755644154) [ 7292.839814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7292.870275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7292.881270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7294.006975] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7294.026192] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7294.085616] Lustre: lustre-MDT0000: new disk, initializing [ 7294.129931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7294.142249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7296.018547] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7301.648193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7301.678897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7301.735294] Lustre: 470887:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7301.777116] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7301.780062] Lustre: Skipped 1 previous similar message [ 7301.822754] Lustre: lustre-MDT0001: new disk, initializing [ 7301.882432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7301.905672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7301.911320] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7303.549545] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7307.613592] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7309.370865] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7312.656512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7312.694881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7312.848449] Lustre: lustre-OST0000: new disk, initializing [ 7312.850749] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7312.893754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7315.422466] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7318.534034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7318.537601] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7318.583067] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7319.330435] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7324.652859] loop0: detected capacity change from 0 to 204800000 [ 7324.746237] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7324.750830] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7324.754875] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7324.758558] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7324.784668] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7324.807209] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7324.823903] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7324.851122] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7324.881066] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7324.912436] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7327.545046] loop0: detected capacity change from 0 to 204800000 [ 7327.681805] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7327.740109] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7329.939470] loop0: detected capacity change from 0 to 204800000 [ 7332.628331] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7332.659488] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7332.734370] Lustre: fs15246-OST0000: new disk, initializing [ 7332.737285] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7332.768355] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7335.223978] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7340.185878] loop1: detected capacity change from 0 to 204800000 [ 7342.895872] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7342.925739] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7342.982466] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7343.024700] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7343.070679] Lustre: fs15246-MDT0000: new disk, initializing [ 7343.143475] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7343.152070] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7343.204585] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7344.226151] Lustre: 475437:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@00000000c799488d x1840926304541312/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1755644225 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7344.660750] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7349.282940] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7349.288451] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7349.298580] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7358.958104] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7358.966436] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7360.779687] LustreError: 476554:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 7360.818938] Lustre: server umount fs15246-MDT0000 complete [ 7365.398572] LustreError: 477055:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 7365.402317] LustreError: 477055:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7365.423754] Lustre: server umount fs15246-OST0000 complete [ 7374.310363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7374.313282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7374.317618] Lustre: Skipped 1 previous similar message [ 7376.434784] LustreError: 477557:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7376.438650] LustreError: 477557:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7376.544582] Lustre: server umount lustre-OST0000 complete [ 7378.913393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7378.917742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7378.925298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7378.928748] Lustre: Skipped 1 previous similar message [ 7384.524979] LustreError: 477759:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7384.528642] LustreError: 477759:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7384.546890] LustreError: 470897:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7384.552682] LustreError: 470897:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 7384.575438] Lustre: server umount lustre-MDT0000 complete [ 7386.368202] LustreError: 470879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755644251 with bad export cookie 7451942195929119045 [ 7386.378087] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7386.605315] Lustre: server umount lustre-MDT0001 complete [ 7391.307069] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 7392.927758] Key type lgssc unregistered [ 7393.124801] LNet: 478486:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7393.129428] LNetError: 478486:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7393.146721] LNet: Removed LNI 192.168.204.132@tcp [ 7393.636701] Key type .llcrypt unregistered [ 7393.638446] Key type ._llcrypt unregistered [ 7402.738189] Key type ._llcrypt registered [ 7402.740143] Key type .llcrypt registered [ 7402.804421] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 7403.166585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7403.178164] alg: No test for adler32 (adler32-zlib) [ 7404.073676] Lustre: Lustre: Build Version: 2.16.57_62_g5cfb5f1 [ 7404.179396] LNet: Added LNI 192.168.204.132@tcp [8/256/0/180] [ 7405.776194] Key type lgssc registered [ 7406.389511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7410.124343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7412.552105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7415.118760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7417.478091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7424.860619] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 7428.951331] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 7433.864484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7437.193658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7440.244253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7443.513459] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7446.982845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7447.017714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7447.027393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7448.163143] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7448.181085] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7448.226681] Lustre: lustre-MDT0000: new disk, initializing [ 7448.255625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7448.265961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7450.130942] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7455.722689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7455.763130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7455.818155] Lustre: 482873:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7455.856247] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7455.859791] Lustre: Skipped 1 previous similar message [ 7455.911854] Lustre: lustre-MDT0001: new disk, initializing [ 7455.963455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7455.985350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7455.992259] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7457.770727] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7461.855762] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7463.673634] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7467.092036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7467.138982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7467.309209] Lustre: lustre-OST0000: new disk, initializing [ 7467.312799] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7467.367987] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7470.107347] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_default_debug -1 all [ 7474.238250] Lustre: DEBUG MARKER: oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7474.706789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7474.713392] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7474.773805] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7476.148597] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7476.263688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7478.171405] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7478.312765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7479.783209] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7479.785111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7479.786693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7479.787475] LustreError: Skipped 1 previous similar message [ 7479.794288] Lustre: Skipped 1 previous similar message [ 7481.269176] LustreError: 485782:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7481.373627] Lustre: server umount lustre-OST0000 complete [ 7484.900316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7484.902271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7484.907271] Lustre: Skipped 1 previous similar message [ 7484.910625] Lustre: Skipped 1 previous similar message [ 7489.356757] LustreError: 485983:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7489.360439] LustreError: 485983:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7489.399814] Lustre: server umount lustre-MDT0000 complete [ 7491.235869] LustreError: 482866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755644356 with bad export cookie 144603309047117493 [ 7491.242784] LustreError: 482866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7491.242984] LustreError: MGC192.168.204.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7491.343507] LustreError: 486184:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7491.349198] LustreError: 486184:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7491.453580] Lustre: server umount lustre-MDT0001 complete [ 7503.035939] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing set_hostid [ 7507.862373] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing load_modules_local [ 7513.323349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7517.286250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7520.459903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7523.894519] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7524.758064] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7394 sec ========== 18:59:49 (1755644389) [ 7525.593866] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:59:50 (1755644390) === [ 7527.026598] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:59:51 (1755644391) === [ 7545.947431] Lustre: DEBUG MARKER: oleg432-server.virtnet: executing unload_modules_local [ 7547.553481] Key type lgssc unregistered [ 7547.743733] LNet: 491019:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7547.749851] LNetError: 491019:0:(acceptor.c:254:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7547.759750] LNet: Removed LNI 192.168.204.132@tcp [ 7548.262330] Key type .llcrypt unregistered [ 7548.264083] Key type ._llcrypt unregistered