[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 451758703 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002248] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004006] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006009] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007005] pid_max: default: 32768 minimum: 301 [ 0.008087] LSM: Security Framework initializing [ 0.009025] Yama: becoming mindful. [ 0.010017] SELinux: Initializing. [ 0.011035] *** VALIDATE selinux *** [ 0.018627] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022644] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023094] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024065] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025057] *** VALIDATE tmpfs *** [ 0.026341] *** VALIDATE proc *** [ 0.027162] *** VALIDATE cgroup *** [ 0.028004] *** VALIDATE cgroup2 *** [ 0.029004] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.030109] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.031003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.032016] Spectre V2 : User space: Vulnerable [ 0.033003] Speculative Store Bypass: Vulnerable [ 0.036057] debug: unmapping init [mem 0xffffffff85e59000-0xffffffff85e60fff] [ 0.038141] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.039427] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.040009] ... version: 2 [ 0.040768] ... bit width: 48 [ 0.041005] ... generic registers: 4 [ 0.041929] ... value mask: 0000ffffffffffff [ 0.042006] ... max period: 00007fffffffffff [ 0.043006] ... fixed-purpose events: 3 [ 0.043874] ... event mask: 000000070000000f [ 0.044222] rcu: Hierarchical SRCU implementation. [ 0.046131] smp: Bringing up secondary CPUs ... [ 0.047451] x86: Booting SMP configuration: [ 0.048014] .... node #0, CPUs: #1 #2 #3 [ 0.051010] smp: Brought up 1 node, 4 CPUs [ 0.052818] smpboot: Max logical packages: 1 [ 0.053008] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.277045] node 0 deferred pages initialised in 223ms [ 0.281228] devtmpfs: initialized [ 0.282181] x86/mm: Memory block size: 128MB [ 0.284210] gcov: version magic: 0x41383552 [ 0.285546] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.286050] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.287160] pinctrl core: initialized pinctrl subsystem [ 0.288093] [ 0.288390] ************************************************************* [ 0.289006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.290006] ** ** [ 0.291005] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.292006] ** ** [ 0.293006] ** This means that this kernel is built to expose internal ** [ 0.294008] ** IOMMU data structures, which may compromise security on ** [ 0.295006] ** your system. ** [ 0.296006] ** ** [ 0.297006] ** If you see this message and you are not debugging the ** [ 0.298006] ** kernel, report this immediately to your vendor! ** [ 0.299006] ** ** [ 0.300006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.301006] ************************************************************* [ 0.302499] NET: Registered protocol family 16 [ 0.303311] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.304026] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.305026] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.306291] cpuidle: using governor menu [ 0.307664] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.311511] PCI: Using configuration type 1 for base access [ 0.313147] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.323058] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.325038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.328143] cryptd: max_cpu_qlen set to 1000 [ 0.332348] ACPI: Added _OSI(Module Device) [ 0.334013] ACPI: Added _OSI(Processor Device) [ 0.336009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.337008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.341983] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.348434] ACPI: Interpreter enabled [ 0.350051] ACPI: PM: (supports S0 S3 S4 S5) [ 0.352008] ACPI: Using IOAPIC for interrupt routing [ 0.355086] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.358393] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.367241] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.369019] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.371008] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.373036] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.377426] acpiphp: Slot [2] registered [ 0.378089] acpiphp: Slot [5] registered [ 0.379021] acpiphp: Slot [6] registered [ 0.379913] acpiphp: Slot [7] registered [ 0.381067] acpiphp: Slot [8] registered [ 0.381895] acpiphp: Slot [9] registered [ 0.383114] acpiphp: Slot [10] registered [ 0.385080] acpiphp: Slot [3] registered [ 0.387065] acpiphp: Slot [4] registered [ 0.389073] acpiphp: Slot [11] registered [ 0.390077] acpiphp: Slot [12] registered [ 0.392079] acpiphp: Slot [13] registered [ 0.394102] acpiphp: Slot [14] registered [ 0.396051] acpiphp: Slot [15] registered [ 0.398071] acpiphp: Slot [16] registered [ 0.400076] acpiphp: Slot [17] registered [ 0.402070] acpiphp: Slot [18] registered [ 0.403072] acpiphp: Slot [19] registered [ 0.404016] acpiphp: Slot [20] registered [ 0.404947] acpiphp: Slot [21] registered [ 0.406045] acpiphp: Slot [22] registered [ 0.406937] acpiphp: Slot [23] registered [ 0.408050] acpiphp: Slot [24] registered [ 0.408973] acpiphp: Slot [25] registered [ 0.409043] acpiphp: Slot [26] registered [ 0.410000] acpiphp: Slot [27] registered [ 0.413074] acpiphp: Slot [28] registered [ 0.414084] acpiphp: Slot [29] registered [ 0.416066] acpiphp: Slot [30] registered [ 0.418083] acpiphp: Slot [31] registered [ 0.419063] PCI host bridge to bus 0000:00 [ 0.421014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.423011] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.426013] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.429013] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.431010] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.434013] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.436180] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.440015] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.443299] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.454842] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.458581] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.462013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.464008] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.468010] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.470610] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.474062] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.477031] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.479748] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.485011] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.499015] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.504017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.510566] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.517013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.523011] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.538020] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.547727] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.554021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.557009] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.568012] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.573917] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.577963] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.581010] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.591008] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.598420] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.604014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.609009] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.620010] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.627045] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.630007] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.634007] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.642008] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.648928] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.652007] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.656015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.664009] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.671920] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.673254] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.674199] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.676212] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.677110] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.681175] iommu: Default domain type: Passthrough [ 0.682296] SCSI subsystem initialized [ 0.684069] ACPI: bus type USB registered [ 0.685055] usbcore: registered new interface driver usbfs [ 0.686028] usbcore: registered new interface driver hub [ 0.687034] usbcore: registered new device driver usb [ 0.689077] pps_core: LinuxPPS API ver. 1 registered [ 0.690006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.692017] PTP clock support registered [ 0.693102] EDAC MC: Ver: 3.0.0 [ 0.695081] PCI: Using ACPI for IRQ routing [ 0.696480] NetLabel: Initializing [ 0.697006] NetLabel: domain hash size = 128 [ 0.697854] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.699048] NetLabel: unlabeled traffic allowed by default [ 0.700068] vgaarb: loaded [ 0.701207] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.703014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.708257] clocksource: Switched to clocksource kvm-clock [ 0.788553] VFS: Disk quotas dquot_6.6.0 [ 0.789567] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.791099] *** VALIDATE ramfs *** [ 0.791896] *** VALIDATE hugetlbfs *** [ 0.792787] pnp: PnP ACPI init [ 0.794347] pnp: PnP ACPI: found 6 devices [ 0.806649] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.810729] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.812672] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.814050] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.815491] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.817284] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.819341] NET: Registered protocol family 2 [ 0.821061] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.824533] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.826760] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.830912] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.833034] TCP: Hash tables configured (established 65536 bind 65536) [ 0.834937] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.837032] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.838961] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.841669] NET: Registered protocol family 1 [ 0.844486] RPC: Registered named UNIX socket transport module. [ 0.845938] RPC: Registered udp transport module. [ 0.847696] RPC: Registered tcp transport module. [ 0.849130] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.850875] NET: Registered protocol family 44 [ 0.852490] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.854254] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.855879] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.858013] PCI: CLS 0 bytes, default 64 [ 0.859418] Unpacking initramfs... [ 2.230929] debug: unmapping init [mem 0xffff9407bcc54000-0xffff9407bffbffff] [ 2.235663] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.237285] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.240269] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.771751] Initialise system trusted keyrings [ 2.773505] Key type blacklist registered [ 2.775383] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.783421] zbud: loaded [ 2.785481] *** VALIDATE nfs *** [ 2.786178] *** VALIDATE nfs4 *** [ 2.787171] pstore: using deflate compression [ 2.788942] Platform Keyring initialized [ 2.858073] NET: Registered protocol family 38 [ 2.859452] Key type asymmetric registered [ 2.860479] Asymmetric key parser 'x509' registered [ 2.862245] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.865687] io scheduler mq-deadline registered [ 2.867911] io scheduler kyber registered [ 2.869944] io scheduler bfq registered [ 2.872081] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.875586] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.878925] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.882351] ACPI: Power Button [PWRF] [ 2.995543] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.099733] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.298867] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.407196] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.605518] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.632901] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.662024] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.673876] Non-volatile memory driver v1.3 [ 3.676761] Linux agpgart interface v0.103 [ 3.706626] virtio_blk virtio1: [vda] 132880 512-byte logical blocks (68.0 MB/64.9 MiB) [ 3.710885] vda: detected capacity change from 0 to 68034560 [ 3.731908] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.735681] vdb: detected capacity change from 0 to 1073741824 [ 3.751349] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.753733] vdc: detected capacity change from 0 to 2621440000 [ 3.765067] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.767583] vdd: detected capacity change from 0 to 2621440000 [ 3.779247] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.781693] vde: detected capacity change from 0 to 4294967296 [ 3.793529] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.795966] vdf: detected capacity change from 0 to 4294967296 [ 3.800989] libphy: Fixed MDIO Bus: probed [ 3.810652] usbcore: registered new interface driver usbserial_generic [ 3.812680] usbserial: USB Serial support registered for generic [ 3.815189] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.820245] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.822567] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.824656] mousedev: PS/2 mouse device common for all mice [ 3.827460] rtc_cmos 00:05: RTC can wake from S4 [ 3.829862] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.833108] rtc_cmos 00:05: registered as rtc0 [ 3.835158] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.835399] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.840097] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.841574] intel_pstate: CPU model not supported [ 3.846423] hid: raw HID events driver (C) Jiri Kosina [ 3.851355] usbcore: registered new interface driver usbhid [ 3.853450] usbhid: USB HID core driver [ 3.855510] drop_monitor: Initializing network drop monitor service [ 3.858271] Initializing XFRM netlink socket [ 3.860406] NET: Registered protocol family 10 [ 3.863726] Segment Routing with IPv6 [ 3.865189] NET: Registered protocol family 17 [ 3.867333] mpls_gso: MPLS GSO support [ 3.872600] RAS: Correctable Errors collector initialized. [ 3.874736] AVX version of gcm_enc/dec engaged. [ 3.876437] AES CTR mode by8 optimization enabled [ 3.949143] sched_clock: Marking stable (3949061523, 0)->(4587455989, -638394466) [ 3.952204] registered taskstats version 1 [ 3.953991] Loading compiled-in X.509 certificates [ 3.955583] zswap: loaded using pool lzo/zbud [ 3.973042] Key type big_key registered [ 3.982615] Key type encrypted registered [ 3.984173] ima: No TPM chip found, activating TPM-bypass! [ 3.985950] ima: Allocated hash algorithm: sha1 [ 3.987516] ima: No architecture policies found [ 3.988939] evm: Initialising EVM extended attributes: [ 3.990550] evm: security.selinux [ 3.991575] evm: security.ima [ 3.992495] evm: security.capability [ 3.993622] evm: HMAC attrs: 0x1 [ 3.995661] rtc_cmos 00:05: setting system clock to 2025-08-20 13:10:26 UTC (1755695426) [ 4.001818] debug: unmapping init [mem 0xffffffff86e03000-0xffffffff86ffffff] [ 4.005786] debug: unmapping init [mem 0xffffffff85b82000-0xffffffff85e58fff] [ 4.015078] Write protecting the kernel read-only data: 28672k [ 4.018025] debug: unmapping init [mem 0xffffffff84203000-0xffffffff843fffff] [ 4.020408] debug: unmapping init [mem 0xffffffff84b14000-0xffffffff84bfffff] [ 4.059225] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.068712] systemd[1]: Detected virtualization kvm. [ 4.070744] systemd[1]: Detected architecture x86-64. [ 4.072889] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.098851] systemd[1]: No hostname configured. [ 4.100394] systemd[1]: Set hostname to . [ 4.102296] random: systemd: uninitialized urandom read (16 bytes read) [ 4.104382] systemd[1]: Initializing machine ID from random generator. [ 4.143686] random: ln: uninitialized urandom read (6 bytes read) [ 4.206073] random: systemd: uninitialized urandom read (16 bytes read) [ 4.207588] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.210289] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.213308] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [[ 4.318380] random: fast init done  OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.690965] device-mapper: uevent: version 1.0.3 [ 4.692754] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.248641] virtio_net virtio0 ens2: renamed from eth0 [ 5.305112] scsi host0: ata_piix [ 5.338532] scsi host1: ata_piix [ 5.339858] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.341997] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.874216] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.498144] random: crng init done [ 10.499369] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.620738] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.619504] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.797678] SELinux: Disabled at runtime. [ 11.854287] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.861841] systemd[1]: Detected virtualization kvm. [ 11.863562] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.205909] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.207952] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.210555] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.212826] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.214580] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.220233] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.225911] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 12.299717] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 12.590438] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.812327] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.862315] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.941229] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.956971] EDAC sbridge: Ver: 1.1.2 [ 14.103109] Key type dns_resolver registered [ 14.429229] NFS: Registering the id_resolver key type [ 14.430649] Key type id_resolver registered [ 14.431674] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg112-server login: [ 36.353161] libcfs: loading out-of-tree module taints kernel. [ 36.361308] Key type ._llcrypt registered [ 36.362147] Key type .llcrypt registered [ 36.399429] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 43.479349] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 44.102962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 44.109996] alg: No test for adler32 (adler32-zlib) [ 45.065257] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 45.364375] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 46.976198] Key type lgssc registered [ 47.775343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 54.088273] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 55.168086] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 58.627900] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 60.995304] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 63.465665] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 68.270020] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 72.816674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 72.838899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 72.849169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 73.942565] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 73.958452] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 74.015149] Lustre: lustre-MDT0000: new disk, initializing [ 74.051755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 74.062234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 75.584583] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 81.378094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 81.424727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 81.465756] Lustre: 6466:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 81.488807] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 81.494493] Lustre: Skipped 1 previous similar message [ 81.542631] Lustre: lustre-MDT0001: new disk, initializing [ 81.568585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 81.579213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 81.585843] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 82.836067] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 85.261634] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 88.970950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 89.010135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 89.120852] Lustre: lustre-OST0000: new disk, initializing [ 89.123472] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 89.142480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 91.310814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 91.314235] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 91.344388] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 91.355284] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 97.448727] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 97.476947] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 97.508419] Lustre: lustre-OST0001: new disk, initializing [ 97.510142] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 97.527135] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 99.439166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 99.443225] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 99.456861] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 99.522425] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 105.472168] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 111.486916] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 118.812593] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing check_logdir /tmp/testlogs/ [ 120.291564] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing yml_node [ 121.708565] Lustre: DEBUG MARKER: Client: 2.16.57.97 [ 122.579745] Lustre: DEBUG MARKER: MDS: 2.16.57.97 [ 123.743743] Lustre: DEBUG MARKER: OSS: 2.16.57.97 [ 124.450141] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 20 09:12:25 EDT 2025 [ 130.916241] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 131.552844] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 133.088420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 133.092445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 133.099060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 135.649385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 135.649755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 135.653031] Lustre: Skipped 1 previous similar message [ 135.662839] Lustre: Skipped 1 previous similar message [ 139.271911] LustreError: 12114:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 139.312995] Lustre: server umount lustre-MDT0000 complete [ 141.047240] LustreError: 6458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695563 with bad export cookie 5771602699669100726 [ 141.048808] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 141.053339] LustreError: 6458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 145.888890] LustreError: 6471:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 145.889162] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 145.889616] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 145.889622] Lustre: Skipped 2 previous similar messages [ 145.896687] LustreError: 6471:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 145.903906] Lustre: Skipped 2 previous similar messages [ 146.912812] LustreError: 7404:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 146.922657] LustreError: 7404:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 147.198410] LustreError: 12315:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 147.201168] LustreError: 12315:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 147.277895] Lustre: server umount lustre-MDT0001 complete [ 148.733627] LustreError: 12517:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 148.737228] LustreError: 12517:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 148.755677] Lustre: server umount lustre-OST0000 complete [ 150.255904] Lustre: server umount lustre-OST0001 complete [ 152.436567] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 155.492673] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 159.008325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 161.750332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 164.722607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 167.463103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 170.096341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 170.130513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 170.256245] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 170.271825] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 170.320657] Lustre: lustre-MDT0000: new disk, initializing [ 170.347920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 170.355269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 171.847155] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 176.729188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 176.768799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 176.824537] Lustre: 14883:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 176.957572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 178.616791] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 182.017354] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 183.461512] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 185.897260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 185.923716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 186.050540] Lustre: lustre-OST0000: new disk, initializing [ 186.051940] Lustre: Skipped 1 previous similar message [ 186.053625] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 186.055741] Lustre: Skipped 2 previous similar messages [ 186.092887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 186.096779] Lustre: Skipped 1 previous similar message [ 187.330812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 187.335507] Lustre: Skipped 1 previous similar message [ 187.339094] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 187.359534] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 188.185817] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 191.689086] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 193.225060] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 193.339331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 194.982394] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 195.101468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 197.603078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 197.607601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 197.610213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 197.614348] Lustre: Skipped 1 previous similar message [ 197.625318] Lustre: Skipped 4 previous similar messages [ 202.723905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 202.727314] Lustre: Skipped 1 previous similar message [ 210.400140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 210.424174] LustreError: 17791:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 210.428052] LustreError: 17791:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 210.519973] Lustre: server umount lustre-OST0000 complete [ 212.961566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 212.962272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 212.965228] Lustre: Skipped 1 previous similar message [ 212.967105] Lustre: Skipped 2 previous similar messages [ 212.969628] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 212.971966] LustreError: Skipped 1 previous similar message [ 218.254686] LustreError: 17995:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 218.257272] LustreError: 17995:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 218.288757] Lustre: server umount lustre-MDT0000 complete [ 219.508038] LustreError: 15658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695642 with bad export cookie 5771602699669102252 [ 219.511557] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 219.513707] LustreError: 15658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 219.651419] Lustre: server umount lustre-MDT0001 complete [ 221.449581] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:14:02 (1755695642) [ 222.514388] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 223.318201] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:14:03 (1755695643) [ 226.312877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 226.482431] LustreError: 18866:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 226.511393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 227.985796] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 231.410302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 232.977926] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 235.108858] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 236.479642] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 239.065385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 241.587912] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 244.175355] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 258.016779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 258.019785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 258.024915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 258.027408] Lustre: Skipped 3 previous similar messages [ 262.119988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 264.113516] LustreError: 21111:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 264.116329] LustreError: 21111:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 264.200215] Lustre: server umount lustre-OST0000 complete [ 271.794698] Lustre: server umount lustre-MDT0000 complete [ 272.353794] LustreError: 18865:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 272.364122] LustreError: 18865:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 273.385723] LustreError: 19577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695695 with bad export cookie 5771602699669103134 [ 273.388451] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 273.391630] LustreError: 19577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 273.535599] Lustre: server umount lustre-MDT0001 complete [ 277.816697] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 279.099653] Key type lgssc unregistered [ 279.241475] LNet: 22042:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 279.244805] LNetError: 22042:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 279.257498] LNet: Removed LNI 192.168.201.112@tcp [ 279.689152] Key type .llcrypt unregistered [ 279.690914] Key type ._llcrypt unregistered [ 288.472306] Key type ._llcrypt registered [ 288.473399] Key type .llcrypt registered [ 288.509321] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 288.836585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 288.870171] alg: No test for adler32 (adler32-zlib) [ 289.712794] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 289.788330] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 291.360265] Key type lgssc registered [ 291.802857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 295.176225] loop: module loaded [ 295.201362] loop0: detected capacity change from 0 to 204800000 [ 295.278251] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 295.282223] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 295.393770] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 295.397897] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 295.419029] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 295.438674] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 295.454937] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 295.486284] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 297.569625] loop0: detected capacity change from 0 to 204800000 [ 297.632579] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 297.638753] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 297.644755] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 297.692710] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 300.103610] loop0: detected capacity change from 0 to 204800000 [ 302.580580] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 302.597751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 302.602668] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 303.681728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 303.692672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 303.728633] Lustre: lustre-MDT0000: new disk, initializing [ 303.752701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 303.759053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 305.172024] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 309.609681] loop1: detected capacity change from 0 to 204800000 [ 312.226748] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 312.255852] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 312.409388] Lustre: lustre-OST07c6: new disk, initializing [ 312.412687] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 312.415601] Lustre: Skipped 1 previous similar message [ 312.445678] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 312.543029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 312.553074] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 312.566162] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 314.961436] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 403.940090] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 403.948304] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 407.091177] LustreError: 26360:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 407.188613] Lustre: server umount lustre-OST07c6 complete [ 411.271814] LustreError: 26860:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 411.275622] LustreError: 26860:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 411.371502] Lustre: server umount lustre-MDT0000 complete [ 418.648728] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 420.071232] Key type lgssc unregistered [ 420.201389] LNet: 27688:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 420.204985] LNetError: 27688:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 420.216450] LNet: Removed LNI 192.168.201.112@tcp [ 420.559439] Key type .llcrypt unregistered [ 420.561562] Key type ._llcrypt unregistered [ 429.625919] Key type ._llcrypt registered [ 429.627122] Key type .llcrypt registered [ 429.658479] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 430.082437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 430.092191] alg: No test for adler32 (adler32-zlib) [ 430.949494] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 431.032462] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 432.608222] Key type lgssc registered [ 433.125481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 436.214967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 436.224265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 437.397241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 438.700828] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 442.284575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 442.500433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 444.214264] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 446.338564] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 447.553570] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 450.185730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 450.326477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 452.657400] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 454.876357] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 461.616810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 468.013219] LustreError: 31110:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 472.035704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 472.038662] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 472.041455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 472.043516] Lustre: Skipped 1 previous similar message [ 477.156215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 477.161386] Lustre: Skipped 2 previous similar messages [ 477.301224] LustreError: 31362:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 477.407589] Lustre: server umount lustre-OST0000 complete [ 482.277125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 482.279206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 482.284773] Lustre: Skipped 1 previous similar message [ 485.064845] LustreError: 31565:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 485.070556] LustreError: 31565:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 485.116749] Lustre: server umount lustre-MDT0000 complete [ 486.967389] LustreError: 29622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695909 with bad export cookie 6736195373173077088 [ 486.973064] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 486.974598] LustreError: 29622:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 487.072242] LustreError: 31766:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 487.075106] LustreError: 31766:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 487.169231] Lustre: server umount lustre-MDT0001 complete [ 491.528279] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 492.927237] Key type lgssc unregistered [ 493.067456] LNet: 32292:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 493.070228] LNetError: 32292:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 493.079346] LNet: Removed LNI 192.168.201.112@tcp [ 493.420945] Key type .llcrypt unregistered [ 493.421981] Key type ._llcrypt unregistered [ 501.088483] Key type ._llcrypt registered [ 501.089620] Key type .llcrypt registered [ 501.120503] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 501.503109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 501.517962] alg: No test for adler32 (adler32-zlib) [ 502.369766] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 502.448943] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 504.024178] Key type lgssc registered [ 504.636256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 508.230547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 508.246966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.425280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 510.875674] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 514.206243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 514.325198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 515.696482] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 517.844475] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 519.377594] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 521.666346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 521.836693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 523.770220] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 525.500556] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 534.127739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 544.744051] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.744463] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 544.745689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 544.752416] Lustre: Skipped 1 previous similar message [ 544.764135] LustreError: Skipped 1 previous similar message [ 545.903087] LustreError: 35762:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 545.958523] Lustre: server umount lustre-OST0000 complete [ 549.857901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 549.858791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 549.861564] Lustre: Skipped 1 previous similar message [ 549.868255] Lustre: Skipped 2 previous similar messages [ 553.489864] LustreError: 35965:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 553.494772] LustreError: 35965:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 553.530461] Lustre: server umount lustre-MDT0000 complete [ 554.977740] LustreError: 33514:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 554.986570] LustreError: 33514:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 555.046909] LustreError: 33495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755695977 with bad export cookie 17119898839596074295 [ 555.050047] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 555.054136] LustreError: 33495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 555.147842] LustreError: 36166:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 555.150823] LustreError: 36166:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 555.241278] Lustre: server umount lustre-MDT0001 complete [ 559.280786] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 560.834883] Key type lgssc unregistered [ 560.961349] LNet: 36692:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 560.963701] LNetError: 36692:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 560.974294] LNet: Removed LNI 192.168.201.112@tcp [ 561.421172] Key type .llcrypt unregistered [ 561.422911] Key type ._llcrypt unregistered [ 572.051377] Key type ._llcrypt registered [ 572.053014] Key type .llcrypt registered [ 572.099655] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 572.510823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 572.517414] alg: No test for adler32 (adler32-zlib) [ 573.382592] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 573.490704] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 575.096132] Key type lgssc registered [ 575.464443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 578.472487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 578.482523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 579.650833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 581.213823] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 584.438649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 584.555670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 586.167044] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 588.346174] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 589.938646] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 592.312839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 592.470094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 594.540533] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 596.727443] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 604.780285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 610.274250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 610.275874] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 610.277620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 610.283640] Lustre: Skipped 1 previous similar message [ 610.286976] Lustre: Skipped 2 previous similar messages [ 611.907306] LustreError: 40309:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 611.938214] Lustre: server umount lustre-MDT0000 complete [ 612.647196] LustreError: 40195:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 612.656946] LustreError: 40195:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 613.503752] LustreError: 38041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696036 with bad export cookie 16650944747567018974 [ 613.510267] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 613.510837] LustreError: 38041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 615.393654] LustreError: 40195:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 615.394147] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 615.411051] Lustre: Skipped 1 previous similar message [ 615.414588] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 617.752295] LustreError: 40195:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 617.767854] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.12@tcp (stopping) [ 619.661079] LustreError: 40511:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 619.665176] LustreError: 40511:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 619.779176] Lustre: server umount lustre-MDT0001 complete [ 620.622617] LustreError: 38041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.12@tcp arrived at 1755696043 with bad export cookie 16650944747567019597 [ 620.630577] LustreError: 38041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 627.691527] LustreError: 40713:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 627.693531] LustreError: 40713:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 627.772880] Lustre: server umount lustre-OST0000 complete [ 634.805735] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 636.152857] Key type lgssc unregistered [ 636.327625] LNet: 41544:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.331587] LNetError: 41544:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 636.341529] LNet: Removed LNI 192.168.201.112@tcp [ 636.714072] Key type .llcrypt unregistered [ 636.715174] Key type ._llcrypt unregistered [ 646.895799] Key type ._llcrypt registered [ 646.897661] Key type .llcrypt registered [ 646.946919] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 647.244738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 647.253556] alg: No test for adler32 (adler32-zlib) [ 648.101975] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 648.192082] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 649.768176] Key type lgssc registered [ 650.190519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 653.376322] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 653.384028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 654.568622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 656.001693] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 659.970635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 660.124621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 661.897131] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 664.581605] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 666.193443] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 669.017271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 669.176354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 670.934929] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 673.402214] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 680.429456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 688.936862] Lustre: lustre-OST0000: Not available for connect from 192.168.201.12@tcp (stopping) [ 690.659162] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 690.660368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 690.667729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 690.669808] Lustre: Skipped 1 previous similar message [ 692.274655] LustreError: 45160:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 692.369607] Lustre: server umount lustre-OST0000 complete [ 695.778775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 695.779823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 695.786415] Lustre: Skipped 2 previous similar messages [ 695.790099] Lustre: Skipped 1 previous similar message [ 700.899132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 700.902942] Lustre: Skipped 1 previous similar message [ 701.647510] LustreError: 45517:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 701.652735] LustreError: 45517:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 701.692095] Lustre: server umount lustre-MDT0000 complete [ 703.559614] LustreError: 42891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696126 with bad export cookie 10572410149187937058 [ 703.566642] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 703.572730] LustreError: 42891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 709.698166] LustreError: 45719:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 709.701456] LustreError: 45719:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 709.792445] Lustre: server umount lustre-MDT0001 complete [ 714.076630] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 715.741325] Key type lgssc unregistered [ 715.932737] LNet: 46246:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 715.937535] LNetError: 46246:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 715.947614] LNet: Removed LNI 192.168.201.112@tcp [ 716.441091] Key type .llcrypt unregistered [ 716.443908] Key type ._llcrypt unregistered [ 727.817111] Key type ._llcrypt registered [ 727.818179] Key type .llcrypt registered [ 727.852294] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 728.110705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 728.134698] alg: No test for adler32 (adler32-zlib) [ 728.975388] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 729.053789] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 730.632172] Key type lgssc registered [ 731.126191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 734.544489] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 734.557539] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 735.741456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 737.233572] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 740.630079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 740.824842] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 742.519681] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 744.951813] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 746.744682] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 749.828334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 749.957926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 752.425790] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 755.048281] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 761.255064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 768.258896] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 769.807599] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 780.835474] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-08-20 9h23m21s [ 781.652671] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 786.213958] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.12@tcp (stopping) [ 786.918137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 786.922216] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 786.925879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 786.931662] Lustre: Skipped 2 previous similar messages [ 786.935467] Lustre: Skipped 1 previous similar message [ 788.006738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 788.010034] Lustre: Skipped 3 previous similar messages [ 788.943829] LustreError: 50181:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 788.987861] Lustre: server umount lustre-MDT0000 complete [ 790.732392] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 792.279664] LustreError: 47610:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 792.293244] LustreError: 47610:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 793.059714] LustreError: 49753:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 793.064222] LustreError: 49753:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 793.723688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 793.783930] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 793.926969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 793.975853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 795.479753] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 796.915755] Lustre: DEBUG MARKER: Wait for df (36017) ... [ 797.612570] Lustre: DEBUG MARKER: done [ 799.202585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 799.202766] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 799.207900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 799.211441] Lustre: Skipped 1 previous similar message [ 800.230208] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 800.811095] LustreError: 51222:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 800.812812] LustreError: 51222:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 800.866600] Lustre: server umount lustre-OST0000 complete [ 805.348048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 805.348590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 805.351194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 805.351202] Lustre: Skipped 2 previous similar messages [ 805.368562] Lustre: Skipped 2 previous similar messages [ 808.205681] LustreError: 51426:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 808.209695] LustreError: 51426:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 808.245891] Lustre: server umount lustre-MDT0000 complete [ 809.797327] LustreError: 47593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696232 with bad export cookie 16769072843479852797 [ 809.800406] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 809.800721] LustreError: 47593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 809.937723] Lustre: server umount lustre-MDT0001 complete [ 814.461266] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 815.750536] Key type lgssc unregistered [ 815.885419] LNet: 52154:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.887948] LNetError: 52154:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.896368] LNet: Removed LNI 192.168.201.112@tcp [ 816.210357] Key type .llcrypt unregistered [ 816.211505] Key type ._llcrypt unregistered [ 828.826618] Key type ._llcrypt registered [ 828.828182] Key type .llcrypt registered [ 828.870186] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 836.370427] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 836.816206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 836.837437] alg: No test for adler32 (adler32-zlib) [ 837.678477] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 837.753601] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 839.336181] Key type lgssc registered [ 839.821970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 843.711103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 846.593616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 849.486241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 851.497782] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 854.353393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 854.389270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 854.401059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 855.504051] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 855.519360] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 855.555507] Lustre: lustre-MDT0000: new disk, initializing [ 855.588492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 855.596549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 856.958746] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 862.427486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 862.465266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 862.515674] Lustre: 56066:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 862.559793] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 862.563358] Lustre: Skipped 1 previous similar message [ 862.608775] Lustre: lustre-MDT0001: new disk, initializing [ 862.656312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 862.678766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 862.684751] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 864.205739] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 867.904420] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 869.543508] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 872.316341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 872.345958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 872.474470] Lustre: lustre-OST0000: new disk, initializing [ 872.476995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 872.510351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 873.466058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 873.468955] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 873.504972] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 874.751185] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 877.946098] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 879.305541] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 879.382390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 880.675153] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 880.769490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 883.685296] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 883.685408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 883.691177] LustreError: Skipped 1 previous similar message [ 883.698560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 883.702889] Lustre: Skipped 1 previous similar message [ 887.663213] LustreError: 58975:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 887.752381] Lustre: server umount lustre-OST0000 complete [ 893.408897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 893.411025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 893.413956] Lustre: Skipped 1 previous similar message [ 893.415576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 895.433842] LustreError: 59176:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 895.436604] LustreError: 59176:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 895.475234] Lustre: server umount lustre-MDT0000 complete [ 896.846415] LustreError: 56058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696319 with bad export cookie 8471004148307606277 [ 896.850268] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 896.851018] LustreError: 56058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 896.924384] LustreError: 59378:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 896.926717] LustreError: 59378:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 897.004503] Lustre: server umount lustre-MDT0001 complete [ 900.187175] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 09:25:20 (1755696320) [ 902.752897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 902.912297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 904.137931] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 907.775759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 907.900861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 909.118730] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 911.261460] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 912.772752] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 915.468799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 917.919726] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 920.015045] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 933.282165] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-08-20 9h25m53s [ 933.944759] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 936.967671] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 937.981719] Lustre: DEBUG MARKER: Creating a test file and stat it [ 938.003784] LustreError: 60046:0:(mdt_handler.c:2329:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 958.242887] Lustre: lustre-MDT0000: Client 9991f8ba-a5d5-4703-a04e-7c8f6eafa8c7 (at 192.168.201.12@tcp) reconnecting [ 978.048173] LustreError: 60046:0:(mdt_handler.c:2329:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 978.829533] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 979.678369] Lustre: DEBUG MARKER: done [ 984.034063] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 984.036799] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 984.040881] Lustre: Skipped 1 previous similar message [ 984.042503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 984.045274] Lustre: Skipped 1 previous similar message [ 987.376100] LustreError: 63056:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 987.381780] LustreError: 63056:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 987.459199] Lustre: server umount lustre-OST0000 complete [ 989.666785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 989.667287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 989.670404] Lustre: Skipped 2 previous similar messages [ 989.676191] Lustre: Skipped 2 previous similar messages [ 995.085289] LustreError: 63258:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 995.089396] LustreError: 63258:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 995.126090] Lustre: server umount lustre-MDT0000 complete [ 995.746186] LustreError: 60757:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 995.752330] LustreError: 60757:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 996.576147] LustreError: 60030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696419 with bad export cookie 8471004148307607159 [ 996.579217] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 996.580178] LustreError: 60030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 996.745774] Lustre: server umount lustre-MDT0001 complete [ 1000.724997] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1001.729108] Key type lgssc unregistered [ 1001.852297] LNet: 63985:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1001.854590] LNetError: 63985:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1001.864247] LNet: Removed LNI 192.168.201.112@tcp [ 1002.160564] Key type .llcrypt unregistered [ 1002.161630] Key type ._llcrypt unregistered [ 1013.304754] Key type ._llcrypt registered [ 1013.305864] Key type .llcrypt registered [ 1013.337487] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 1020.497667] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1020.894666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1020.906791] alg: No test for adler32 (adler32-zlib) [ 1021.756796] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 1021.834390] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1023.408175] Key type lgssc registered [ 1023.847249] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1026.921650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1029.613074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1031.826592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1034.184041] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1036.974272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1036.998919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1037.004491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1038.086765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1038.100336] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1038.138037] Lustre: lustre-MDT0000: new disk, initializing [ 1038.163324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1038.170248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1039.763811] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1044.560482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1044.585447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1044.621702] Lustre: 67895:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1044.651835] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1044.654437] Lustre: Skipped 1 previous similar message [ 1044.691381] Lustre: lustre-MDT0001: new disk, initializing [ 1044.730392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1044.744435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1044.749307] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1046.125715] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1049.690719] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1050.904638] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1053.356530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1053.388950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1053.499765] Lustre: lustre-OST0000: new disk, initializing [ 1053.501987] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1053.533873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1055.421495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1055.424727] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1055.464314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1055.843923] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1059.342599] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1060.857370] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1060.923542] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1062.576758] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1062.653501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1065.956449] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1065.960335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1065.961049] LustreError: Skipped 1 previous similar message [ 1065.966163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1065.969436] Lustre: Skipped 1 previous similar message [ 1069.618226] LustreError: 70802:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1069.721947] Lustre: server umount lustre-OST0000 complete [ 1075.681316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1075.683396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1075.686566] Lustre: Skipped 1 previous similar message [ 1075.688346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1077.459877] LustreError: 71004:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1077.463635] LustreError: 71004:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1077.499812] Lustre: server umount lustre-MDT0000 complete [ 1078.856397] LustreError: 69628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696501 with bad export cookie 12517104786043235452 [ 1078.859277] LustreError: 69628:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1078.859929] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1078.924343] LustreError: 71209:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1078.926188] LustreError: 71209:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1079.023750] Lustre: server umount lustre-MDT0001 complete [ 1082.577564] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 09:28:23 (1755696503) [ 1083.159447] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1083.928607] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 09:28:24 (1755696504) [ 1088.457784] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1090.052229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1090.242696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1090.605379] Lustre: Failing over lustre-MDT0000 [ 1090.606923] LustreError: 72277:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1090.611453] LustreError: 72277:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1090.614784] LustreError: 72277:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 1090.682063] LustreError: 72328:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1090.683812] LustreError: 72328:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1090.775340] Lustre: server umount lustre-MDT0000 complete [ 1093.061142] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:28:33 (1755696513) [ 1095.623361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1095.788919] LustreError: 72896:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1095.797138] LustreError: 72896:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1095.806364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1097.188684] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1100.399773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1101.969464] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1103.872648] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1105.266750] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1108.025278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1108.152798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.156168] Lustre: Skipped 1 previous similar message [ 1110.210444] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1112.014100] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1125.140380] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1131.489443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1131.494303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1131.501630] Lustre: Skipped 1 previous similar message [ 1131.505091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1131.508896] Lustre: Skipped 2 previous similar messages [ 1133.694823] LustreError: 75243:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1133.697135] LustreError: 75243:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1133.728502] Lustre: server umount lustre-MDT0000 complete [ 1135.011316] LustreError: 72896:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1135.015670] LustreError: 72896:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1135.327590] LustreError: 72878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696557 with bad export cookie 12517104786043236593 [ 1135.330818] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1135.331736] LustreError: 72878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1135.502864] Lustre: server umount lustre-MDT0001 complete [ 1136.775329] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1138.459790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1142.293741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1145.312926] LustreError: 66109:0:(client.c:1385:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cd065483 x1840981154522624/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1145.471798] LustreError: 74491:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1145.509310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1145.541432] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1146.848714] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1149.776293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1149.886977] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1149.891576] Lustre: Skipped 2 previous similar messages [ 1151.247349] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1153.065189] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1154.253379] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1155.048195] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1155.680231] Lustre: 66113:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1755696562/real 1755696562] req@0000000094df3cee x1840981154522368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1755696578 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1160.167222] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1161.168164] Lustre: 66113:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1755696567/real 1755696567] req@00000000f7f4a658 x1840981154522496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1755696583 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1165.283188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1165.284068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1165.285090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1165.285097] Lustre: Skipped 2 previous similar messages [ 1165.286868] Lustre: Skipped 2 previous similar messages [ 1167.107865] LustreError: 77692:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1167.109587] LustreError: 77692:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1167.145276] Lustre: server umount lustre-MDT0000 complete [ 1168.508959] LustreError: 72877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696591 with bad export cookie 12517104786043238483 [ 1168.513563] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1168.516731] LustreError: 72877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1168.683378] Lustre: server umount lustre-MDT0001 complete [ 1171.386667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1174.846789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1178.529040] LustreError: 66109:0:(client.c:1385:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005e47eab3 x1840981154542080/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1178.720062] LustreError: 74490:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1178.748213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1178.750110] Lustre: Skipped 1 previous similar message [ 1178.766755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1179.996610] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1182.835743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1182.932926] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1184.450828] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1184.998668] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1186.016108] Lustre: 66113:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1755696592/real 1755696592] req@000000000690347e x1840981154541696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1755696608 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1186.313846] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1188.004400] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1189.088155] Lustre: 66111:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1755696595/real 1755696595] req@000000003502044d x1840981154541824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1755696611 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1190.375404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1190.379050] Lustre: Skipped 4 previous similar messages [ 1195.488328] Lustre: 66112:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1755696601/real 1755696601] req@000000006a40112c x1840981154542208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1755696617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1195.502080] Lustre: 66112:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1195.979180] LustreError: 80133:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1195.983208] LustreError: 80133:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1196.025313] Lustre: server umount lustre-MDT0000 complete [ 1197.504120] LustreError: 72876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696620 with bad export cookie 12517104786043239855 [ 1197.510385] LustreError: 72876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1197.510953] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1197.694695] Lustre: server umount lustre-MDT0001 complete [ 1200.055263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1201.341169] Lustre: DEBUG MARKER: files compared the same [ 1212.528965] Lustre: server umount lustre-OST0000 complete [ 1218.006557] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1219.207169] Key type lgssc unregistered [ 1219.341632] LNet: 81758:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1219.345202] LNetError: 81758:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1219.356537] LNet: Removed LNI 192.168.201.112@tcp [ 1219.691989] Key type .llcrypt unregistered [ 1219.693130] Key type ._llcrypt unregistered [ 1228.495550] Key type ._llcrypt registered [ 1228.496713] Key type .llcrypt registered [ 1228.534284] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1228.895016] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1228.933807] alg: No test for adler32 (adler32-zlib) [ 1229.777455] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 1229.856804] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1231.432227] Key type lgssc registered [ 1231.817899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1234.426083] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1234.435297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1235.591595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1237.096979] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug +malloc all [ 1239.915871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1240.011993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1241.368720] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug +malloc all [ 1243.515502] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1244.887097] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1247.626497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1247.729848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1249.969553] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug +malloc all [ 1251.853716] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1261.030787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1265.121052] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1265.125473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1265.132175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1266.146240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1266.147281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1266.158839] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1271.022116] LustreError: 85375:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1271.111318] Lustre: server umount lustre-OST0000 complete [ 1275.872999] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1275.876654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1275.882543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1275.887437] Lustre: Skipped 1 previous similar message [ 1278.722261] LustreError: 85578:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1278.724466] LustreError: 85578:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1278.760306] Lustre: server umount lustre-MDT0000 complete [ 1280.315150] LustreError: 83106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696702 with bad export cookie 7988183956033639092 [ 1280.317509] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1280.327197] LustreError: 83106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1280.418988] LustreError: 85779:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1280.422727] LustreError: 85779:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1280.500579] Lustre: server umount lustre-MDT0001 complete [ 1284.920129] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1286.045121] Key type lgssc unregistered [ 1286.171329] LNet: 86304:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1286.173677] LNetError: 86304:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1286.184273] LNet: Removed LNI 192.168.201.112@tcp [ 1286.479635] Key type .llcrypt unregistered [ 1286.480651] Key type ._llcrypt unregistered [ 1295.439385] Key type ._llcrypt registered [ 1295.440498] Key type .llcrypt registered [ 1295.479150] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1295.871515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1295.933711] alg: No test for adler32 (adler32-zlib) [ 1296.801698] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 1296.898801] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1298.488145] Key type lgssc registered [ 1298.913734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1301.847366] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1301.853758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1318.304396] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1324.448292] LustreError: 87645:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.112@tcp: failed processing log, type 4: rc = -110 [ 1354.144164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1356.069935] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1357.837751] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1360.743379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1360.802562] Lustre: *** cfs_fail_loc=706, val=0*** [ 1361.000285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1361.039537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1362.340977] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1366.162727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1366.301441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1367.566665] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1369.306991] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1370.438748] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1371.617295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1371.620553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1371.626502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1373.230983] LustreError: 89948:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1373.316539] Lustre: server umount lustre-OST0000 complete [ 1376.737288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1376.740598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1376.754755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1379.810897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1379.811626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1384.930965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1384.935263] Lustre: Skipped 2 previous similar messages [ 1384.935727] LustreError: 90152:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1384.941132] LustreError: 90152:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1384.976561] Lustre: server umount lustre-MDT0000 complete [ 1386.767188] LustreError: 89950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755696809 with bad export cookie 11478235630840588634 [ 1386.770779] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1386.771773] LustreError: 89950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1386.860314] LustreError: 90353:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1386.863209] LustreError: 90353:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1387.000955] Lustre: server umount lustre-MDT0001 complete [ 1392.682062] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1394.251815] Key type lgssc unregistered [ 1394.413392] LNet: 90880:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1394.417529] LNetError: 90880:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1394.426896] LNet: Removed LNI 192.168.201.112@tcp [ 1394.811916] Key type .llcrypt unregistered [ 1394.813543] Key type ._llcrypt unregistered [ 1404.076387] Key type ._llcrypt registered [ 1404.077911] Key type .llcrypt registered [ 1404.119567] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1404.486252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1404.509338] alg: No test for adler32 (adler32-zlib) [ 1405.362070] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 1405.439843] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1407.024148] Key type lgssc registered [ 1407.384040] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1410.035316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1410.043794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1412.367798] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1414.893344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1415.075198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1416.337258] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1419.279934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1419.406229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.384823] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1424.311640] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1424.317990] Lustre: MGS: Client b0d62bed-14fb-4f08-954b-0455ec19ff72 (at 0@lo) reconnecting [ 1424.411500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1424.444561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1425.762777] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1434.593315] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1434.596221] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1434.602123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1435.619516] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1435.620114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1435.620962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1435.631877] Lustre: Skipped 1 previous similar message [ 1439.661087] LustreError: 94855:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1439.720212] Lustre: server umount lustre-OST0000 complete [ 1445.858048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1445.858151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.858797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1445.869881] Lustre: Skipped 1 previous similar message [ 1447.036370] LustreError: 95057:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1447.038516] LustreError: 95057:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1447.046830] Lustre: server umount lustre-MDT0000 complete [ 1448.345743] LustreError: 95259:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1448.347866] LustreError: 95259:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1448.419917] Lustre: server umount lustre-MDT0001 complete [ 1449.751668] Lustre: server umount MGS complete [ 1453.576416] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:34:34 (1755696874) [ 1462.945719] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 1466.172539] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1469.582518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1471.777983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1473.870416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1476.044829] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1478.400841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1478.427302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1479.714989] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1483.679095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1483.699377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1483.775512] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1483.792886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1483.829584] Lustre: lustre-MDT0001: new disk, initializing [ 1483.854450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1483.863275] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1483.865686] Lustre: 99947:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1483.868123] LustreError: 99947:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1485.180250] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1489.349853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1489.379531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1489.492410] Lustre: lustre-OST0000: new disk, initializing [ 1489.496104] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1489.524094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1491.506270] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1494.701602] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1496.096897] Lustre: 101304:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@00000000c8fd0fda x1840981556102400/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1755696934 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1497.073720] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1497.078880] Lustre: MGS: Client 2ca163f3-3135-43b1-b715-78783ebf69b7 (at 0@lo) reconnecting [ 1497.091762] Lustre: 99239:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1497.101952] LustreError: 99239:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1497.112787] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1497.150814] Lustre: lustre-MDT0000: new disk, initializing [ 1497.208234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1498.272923] Lustre: 101647:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@000000002b7b22ef x1840981556107648/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1755696936 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1498.273506] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1498.641074] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1503.331852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1503.336455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1503.340353] Lustre: Skipped 1 previous similar message [ 1503.344444] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1512.419074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1512.420533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1512.422884] Lustre: Skipped 1 previous similar message [ 1512.424840] Lustre: Skipped 1 previous similar message [ 1517.538923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1517.541442] Lustre: Skipped 2 previous similar messages [ 1518.317100] LustreError: 102165:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1518.320666] LustreError: 102165:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1518.389674] Lustre: server umount lustre-OST0000 complete [ 1522.657583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1522.657895] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1522.664677] Lustre: Skipped 1 previous similar message [ 1525.885096] LustreError: 102368:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1525.886943] LustreError: 102368:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1525.895996] Lustre: server umount lustre-MDT0000 complete [ 1533.316894] Lustre: server umount lustre-MDT0001 complete [ 1534.555205] LustreError: 102771:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1534.557039] LustreError: 102771:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1534.580385] Lustre: server umount MGS complete [ 1538.552876] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:35:59 (1755696959) [ 1544.318349] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1545.363414] Key type lgssc unregistered [ 1545.493336] LNet: 104101:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1545.495448] LNetError: 104101:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1545.507300] LNet: Removed LNI 192.168.201.112@tcp [ 1545.816225] Key type .llcrypt unregistered [ 1545.817551] Key type ._llcrypt unregistered [ 1552.535221] Key type ._llcrypt registered [ 1552.537044] Key type .llcrypt registered [ 1552.572074] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1552.914346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1552.976498] alg: No test for adler32 (adler32-zlib) [ 1553.841536] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 1553.933366] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1555.528160] Key type lgssc registered [ 1556.048754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1559.165150] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1559.173899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1560.360710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1561.674349] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1565.259701] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1565.417798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1566.987295] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1569.002466] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1570.241148] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1572.619626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1572.751436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1574.635949] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1576.480844] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1585.068071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1590.241781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1590.242893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1590.244339] LustreError: Skipped 1 previous similar message [ 1590.251683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1590.255294] Lustre: Skipped 1 previous similar message [ 1591.854176] LustreError: 107524:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1591.914389] Lustre: server umount lustre-OST0000 complete [ 1595.262744] LustreError: 107724:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1595.264516] LustreError: 107724:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1595.289709] Lustre: server umount lustre-MDT0000 complete [ 1595.362748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1595.363279] LustreError: 105277:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1595.368111] Lustre: Skipped 2 previous similar messages [ 1595.372104] LustreError: 105277:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1596.433790] LustreError: 105258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697018 with bad export cookie 17624057513576292989 [ 1596.438974] LustreError: 105258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1596.439102] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1596.502108] LustreError: 107927:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1596.503856] LustreError: 107927:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1596.592605] Lustre: server umount lustre-MDT0001 complete [ 1600.071755] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1601.333596] Key type lgssc unregistered [ 1601.487703] LNet: 108452:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1601.492277] LNetError: 108452:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1601.504645] LNet: Removed LNI 192.168.201.112@tcp [ 1601.811148] Key type .llcrypt unregistered [ 1601.812229] Key type ._llcrypt unregistered [ 1609.931817] Key type ._llcrypt registered [ 1609.932807] Key type .llcrypt registered [ 1609.971645] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1610.401749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1610.443681] alg: No test for adler32 (adler32-zlib) [ 1611.294818] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 1611.376945] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1612.960180] Key type lgssc registered [ 1613.295581] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1615.401493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1615.401666] LustreError: 109485:0:(tgt_mount.c:2416:server_fill_super()) cfs_race id 716 sleeping [ 1615.403871] Lustre: Skipped 1 previous similar message [ 1615.406947] LustreError: 109486:0:(tgt_mount.c:2416:server_fill_super()) cfs_fail_race id 716 waking [ 1615.408939] LustreError: 109485:0:(tgt_mount.c:2416:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 1615.411086] LustreError: 109485:0:(tgt_mount.c:2428:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1615.415078] LustreError: 109485:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1615.415618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1616.616001] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1619.693344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1619.802608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1621.279473] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1622.971443] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing lsmod [ 1623.932098] LustreError: 110652:0:(tgt_mount.c:2416:server_fill_super()) cfs_race id 716 sleeping [ 1623.932797] LustreError: 110651:0:(tgt_mount.c:2416:server_fill_super()) cfs_fail_race id 716 waking [ 1623.936938] LustreError: 110652:0:(tgt_mount.c:2416:server_fill_super()) cfs_fail_race id 716 awake: rc=4998 [ 1623.938924] LustreError: 110652:0:(tgt_mount.c:2428:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1623.942930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1623.943152] LustreError: 110652:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1624.079894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1625.058654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1625.063589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1625.070927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1629.154336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1629.156977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1629.160663] Lustre: Skipped 1 previous similar message [ 1630.180641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1630.182697] Lustre: Skipped 4 previous similar messages [ 1635.299762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1635.305512] Lustre: Skipped 2 previous similar messages [ 1639.392070] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1639.426542] LustreError: 110839:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1639.458442] Lustre: server umount lustre-MDT0000 complete [ 1640.419951] LustreError: 109521:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1640.428600] LustreError: 109521:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 1640.777887] LustreError: 109504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697063 with bad export cookie 18214096609426851317 [ 1640.782132] LustreError: 109504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1640.782717] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1642.464322] LustreError: 109959:0:(client.c:1375:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000fb7d6976 x1840981772092416/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1642.484107] LustreError: 111054:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1642.488231] LustreError: 111054:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1642.580676] Lustre: server umount lustre-MDT0001 complete [ 1653.748436] LustreError: 111255:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1653.750252] LustreError: 111255:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1653.838268] Lustre: server umount lustre-OST0000 complete [ 1657.018257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1657.223637] LustreError: 111732:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1657.254422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1658.988565] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1662.171707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1662.296314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1663.700447] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1665.886419] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1667.168903] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1669.949966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1672.423326] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1674.366743] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1682.421337] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1687.523366] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1687.524473] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1687.525263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1687.529503] Lustre: Skipped 1 previous similar message [ 1687.532969] LustreError: Skipped 1 previous similar message [ 1688.173288] LustreError: 113980:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1688.176133] LustreError: 113980:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1688.248688] Lustre: server umount lustre-OST0000 complete [ 1691.898647] Lustre: server umount lustre-MDT0000 complete [ 1692.642319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1692.643417] LustreError: 112985:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1692.648037] Lustre: Skipped 1 previous similar message [ 1692.656751] LustreError: 112985:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1693.750794] LustreError: 111712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697116 with bad export cookie 18214096609426851975 [ 1693.755208] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1693.760813] LustreError: 111712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1693.845247] LustreError: 114384:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1693.847923] LustreError: 114384:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 1693.973168] Lustre: server umount lustre-MDT0001 complete [ 1698.644477] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1700.043441] Key type lgssc unregistered [ 1700.221521] LNet: 114910:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1700.225977] LNetError: 114910:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1700.240951] LNet: Removed LNI 192.168.201.112@tcp [ 1700.696088] Key type .llcrypt unregistered [ 1700.697592] Key type ._llcrypt unregistered [ 1712.957276] Key type ._llcrypt registered [ 1712.958903] Key type .llcrypt registered [ 1713.014867] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1713.525206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1713.534921] alg: No test for adler32 (adler32-zlib) [ 1714.495853] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 1714.623064] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1716.232140] Key type lgssc registered [ 1716.761296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1720.689430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1720.705119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1721.988641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1723.848388] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1727.954938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1728.138806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1729.786722] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1732.165305] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1733.757361] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1736.260375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1736.378327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1738.439738] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1740.437338] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1746.599833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 1752.731408] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1753.506417] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1757.155120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1757.155847] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1757.156483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1757.160965] Lustre: Skipped 1 previous similar message [ 1760.492818] LustreError: 118627:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1760.556677] Lustre: server umount lustre-OST0000 complete [ 1762.274107] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1762.274835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1762.278029] Lustre: Skipped 1 previous similar message [ 1762.280171] Lustre: Skipped 1 previous similar message [ 1767.392891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1767.394728] Lustre: Skipped 2 previous similar messages [ 1767.999981] LustreError: 118833:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1768.002160] LustreError: 118833:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1768.032806] Lustre: server umount lustre-MDT0000 complete [ 1769.387309] LustreError: 118480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697191 with bad export cookie 10257447113829405398 [ 1769.389690] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1769.391792] LustreError: 118480:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1769.460941] LustreError: 119036:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1769.463922] LustreError: 119036:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1769.551522] Lustre: server umount lustre-MDT0001 complete [ 1773.721738] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1774.896728] Key type lgssc unregistered [ 1775.038386] LNet: 119562:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1775.041709] LNetError: 119562:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1775.054320] LNet: Removed LNI 192.168.201.112@tcp [ 1775.362686] Key type .llcrypt unregistered [ 1775.364394] Key type ._llcrypt unregistered [ 1782.529497] Key type ._llcrypt registered [ 1782.531146] Key type .llcrypt registered [ 1782.569252] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1782.848714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1782.873327] alg: No test for adler32 (adler32-zlib) [ 1783.714869] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 1783.793856] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1785.368187] Key type lgssc registered [ 1785.716558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1788.362026] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1788.368801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1789.523881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1790.858095] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1793.828483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1793.968209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1795.288369] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1797.191054] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1798.425722] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1800.669023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1800.796387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1802.547956] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1804.475818] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1815.148487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 1824.738321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1824.740645] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1824.745061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1825.762791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1825.765757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1825.766376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1825.771814] Lustre: Skipped 1 previous similar message [ 1828.589108] LustreError: 122983:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1828.654684] Lustre: server umount lustre-OST0000 complete [ 1830.881787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1830.882776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1830.885650] Lustre: Skipped 1 previous similar message [ 1830.891068] Lustre: Skipped 1 previous similar message [ 1836.000987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1836.003506] Lustre: Skipped 1 previous similar message [ 1836.158505] LustreError: 123185:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1836.160879] LustreError: 123185:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1836.183757] Lustre: server umount lustre-MDT0000 complete [ 1837.527246] LustreError: 120715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697260 with bad export cookie 1120831845335917459 [ 1837.530673] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1837.532192] LustreError: 120715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1837.598000] LustreError: 123387:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1837.601116] LustreError: 123387:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1837.689652] Lustre: server umount lustre-MDT0001 complete [ 1841.643542] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1842.841200] Key type lgssc unregistered [ 1842.977416] LNet: 123914:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1842.980826] LNetError: 123914:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1842.993544] LNet: Removed LNI 192.168.201.112@tcp [ 1843.326974] Key type .llcrypt unregistered [ 1843.328218] Key type ._llcrypt unregistered [ 1853.144060] Key type ._llcrypt registered [ 1853.145321] Key type .llcrypt registered [ 1853.190170] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1853.570485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1853.586051] alg: No test for adler32 (adler32-zlib) [ 1854.464138] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 1854.559629] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1856.160223] Key type lgssc registered [ 1856.589367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1859.734621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1859.742224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1860.948235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1862.471482] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1865.696820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1865.854860] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1867.377184] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1869.465949] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1870.844088] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1873.527232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1873.659611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1875.836784] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1878.019615] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1885.930926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 1891.144637] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1892.509115] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1892.512064] Lustre: Skipped 2 previous similar messages [ 1894.069977] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1894.073772] Lustre: Skipped 2 previous similar messages [ 1897.982358] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1897.994318] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1903.529896] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.12@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1903.533211] Lustre: Skipped 2 previous similar messages [ 1903.783190] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.12@tcp 192.168.0.[2,10]@tcp [ 1909.209649] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.12/24@tcp in log lustre-MDT0000 [ 1909.212553] Lustre: Skipped 2 previous similar messages [ 1912.200591] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.12/24@tcp [ 1912.203780] Lustre: Skipped 1 previous similar message [ 1922.018081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1922.019511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1922.026630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1922.028473] Lustre: Skipped 1 previous similar message [ 1924.973113] LustreError: 128842:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1925.031875] Lustre: server umount lustre-OST0000 complete [ 1927.138298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1927.139204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1927.142373] Lustre: Skipped 2 previous similar messages [ 1932.257587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1932.260225] Lustre: Skipped 2 previous similar messages [ 1932.412612] LustreError: 129043:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1932.414664] LustreError: 129043:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1932.440117] Lustre: server umount lustre-MDT0000 complete [ 1933.630728] LustreError: 127587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697356 with bad export cookie 10276587474922987446 [ 1933.634830] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1933.698663] LustreError: 129244:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1933.700404] LustreError: 129244:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1933.780829] Lustre: server umount lustre-MDT0001 complete [ 1937.423187] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 1938.558380] Key type lgssc unregistered [ 1938.693398] LNet: 129769:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1938.696140] LNetError: 129769:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1938.702291] LNet: Removed LNI 192.168.201.112@tcp [ 1939.013571] Key type .llcrypt unregistered [ 1939.014601] Key type ._llcrypt unregistered [ 1949.425503] Key type ._llcrypt registered [ 1949.427116] Key type .llcrypt registered [ 1949.474806] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 1949.767491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1949.774457] alg: No test for adler32 (adler32-zlib) [ 1950.653930] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 1950.740500] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 1952.328180] Key type lgssc registered [ 1952.729302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1955.463649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1955.473587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1956.654396] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1956.660783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1956.667085] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.12/24@tcp [ 1958.107322] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1961.451656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1961.604529] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1961.617923] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.12/24@tcp [ 1961.638599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1963.096736] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1965.107973] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1966.447804] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1968.949386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1969.072433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1971.174711] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 1973.181631] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1982.377578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 1987.555294] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1987.555786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1987.558527] LustreError: Skipped 1 previous similar message [ 1987.565604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1987.570504] Lustre: Skipped 1 previous similar message [ 1989.295173] LustreError: 133585:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1989.388438] Lustre: server umount lustre-OST0000 complete [ 1992.674076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1992.674251] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1992.674923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1992.684443] Lustre: Skipped 2 previous similar messages [ 1993.092708] LustreError: 133787:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1993.094850] LustreError: 133787:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1993.127909] Lustre: server umount lustre-MDT0000 complete [ 1994.527260] LustreError: 131262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697417 with bad export cookie 14150554365658765024 [ 1994.530316] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1994.532131] LustreError: 131262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1994.605576] LustreError: 133987:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1994.608560] LustreError: 133987:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1994.698593] Lustre: server umount lustre-MDT0001 complete [ 1998.884794] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 2000.031099] Key type lgssc unregistered [ 2000.168469] LNet: 134514:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2000.171629] LNetError: 134514:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2000.180327] LNet: Removed LNI 192.168.201.112@tcp [ 2000.512622] Key type .llcrypt unregistered [ 2000.514127] Key type ._llcrypt unregistered [ 2009.501383] Key type ._llcrypt registered [ 2009.503084] Key type .llcrypt registered [ 2009.541695] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 2016.010331] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2016.447986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2016.454948] alg: No test for adler32 (adler32-zlib) [ 2017.317290] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 2017.405280] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 2018.984159] Key type lgssc registered [ 2019.357319] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2022.429502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2024.712830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2026.682330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2028.792123] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2031.064216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2031.086508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2031.091664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2032.169593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2032.181013] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2032.215622] Lustre: lustre-MDT0000: new disk, initializing [ 2032.241124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2032.248130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2033.540172] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2037.791077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2037.817241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2037.850762] Lustre: 137848:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2037.875328] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2037.878384] Lustre: Skipped 1 previous similar message [ 2037.918343] Lustre: lustre-MDT0001: new disk, initializing [ 2037.950319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2037.964718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2037.968855] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2039.305969] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2042.526688] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2043.872583] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2046.337074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2046.359219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2046.452046] Lustre: lustre-OST0000: new disk, initializing [ 2046.453885] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2046.483981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2048.484131] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2048.758934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2048.762958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2048.800283] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2051.759525] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2053.100590] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2053.182118] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2054.399225] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2054.477109] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2057.134621] LustreError: 140757:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2057.197406] Lustre: server umount lustre-OST0000 complete [ 2058.721242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2058.723506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2058.728153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2059.233491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2059.233829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2064.353388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2064.355285] Lustre: Skipped 2 previous similar messages [ 2068.452266] LustreError: 140957:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2068.454183] LustreError: 140957:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2068.480874] Lustre: server umount lustre-MDT0000 complete [ 2069.750183] LustreError: 138621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697492 with bad export cookie 6316825649401975466 [ 2069.753584] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2069.755762] LustreError: 138621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2080.228304] LustreError: 141159:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2080.230357] LustreError: 141159:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2080.310212] Lustre: server umount lustre-MDT0001 complete [ 2083.339024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2083.518627] LustreError: 141637:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2083.524705] LustreError: 141637:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2083.546549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2084.951942] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2087.990625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2088.096047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2089.306593] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2091.067183] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2092.217328] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2094.414317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2096.415450] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2098.212508] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2099.526722] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2099.600667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2100.823901] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2100.898396] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2104.186057] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2104.207067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2104.263736] Lustre: lustre-OST0001: new disk, initializing [ 2104.266750] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2104.300991] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2104.303616] Lustre: Skipped 1 previous similar message [ 2105.284023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2105.287090] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2105.318108] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2106.232473] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2109.394721] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2109.465313] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2110.772396] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2110.849902] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2112.117436] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa01747d7f000.ost_server_uuid 50 [ 2112.631149] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa01747d7f000.ost_server_uuid in FULL state after 0 sec [ 2115.556744] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2115.556912] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2115.557798] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2115.562046] Lustre: Skipped 1 previous similar message [ 2115.566978] LustreError: Skipped 1 previous similar message [ 2116.460510] LustreError: 145836:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2116.462827] LustreError: 145836:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2116.480118] Lustre: server umount lustre-OST0001 complete [ 2118.112921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2118.115395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2119.777354] Lustre: server umount lustre-OST0000 complete [ 2124.256759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2124.259413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2124.263568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2124.265338] Lustre: Skipped 2 previous similar messages [ 2130.916388] LustreError: 146237:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2130.918152] LustreError: 146237:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2130.942205] Lustre: server umount lustre-MDT0000 complete [ 2132.115425] LustreError: 141618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697554 with bad export cookie 6316825649401976334 [ 2132.118608] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2132.118886] LustreError: 141618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2132.265477] Lustre: server umount lustre-MDT0001 complete [ 2138.481655] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 2139.555589] Key type lgssc unregistered [ 2139.700369] LNet: 147423:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2139.703497] LNetError: 147423:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2139.711327] LNet: Removed LNI 192.168.201.112@tcp [ 2140.009863] Key type .llcrypt unregistered [ 2140.011637] Key type ._llcrypt unregistered [ 2151.112949] Key type ._llcrypt registered [ 2151.114800] Key type .llcrypt registered [ 2151.153872] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 2157.323398] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2157.746258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2157.754241] alg: No test for adler32 (adler32-zlib) [ 2158.617844] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 2158.702939] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 2160.296181] Key type lgssc registered [ 2160.653248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2163.548132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2165.792351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2167.767350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2169.768228] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2172.102960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2172.122357] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2172.129147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2173.224616] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2173.236309] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2173.272840] Lustre: lustre-MDT0000: new disk, initializing [ 2173.306755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2173.316478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2174.674555] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2178.817074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2178.840823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2178.875894] Lustre: 151331:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2178.905389] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2178.907485] Lustre: Skipped 1 previous similar message [ 2178.942625] Lustre: lustre-MDT0001: new disk, initializing [ 2178.981609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2178.994530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2178.997917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2180.297499] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2183.467359] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2184.768745] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2187.015867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2187.044169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2187.138630] Lustre: lustre-OST0000: new disk, initializing [ 2187.141679] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2187.169442] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2187.958611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2187.963141] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2187.995127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2188.930289] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2191.986161] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2193.333807] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2193.404826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2194.713643] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2194.787140] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2198.497359] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2198.498430] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2198.499972] LustreError: Skipped 1 previous similar message [ 2198.503681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2198.506088] Lustre: Skipped 1 previous similar message [ 2201.453116] LustreError: 154239:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2201.531470] Lustre: server umount lustre-OST0000 complete [ 2203.618222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2203.619152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2203.622313] Lustre: Skipped 2 previous similar messages [ 2208.737787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2208.739924] Lustre: Skipped 2 previous similar messages [ 2208.893766] LustreError: 154440:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2208.896882] LustreError: 154440:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2208.923964] Lustre: server umount lustre-MDT0000 complete [ 2210.177212] LustreError: 151324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697632 with bad export cookie 10541079713859508472 [ 2210.179362] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2210.181748] LustreError: 151324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2210.239152] LustreError: 154643:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2210.242017] LustreError: 154643:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2210.322085] Lustre: server umount lustre-MDT0001 complete [ 2213.131163] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 09:47:13 (1755697633) [ 2218.118404] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 2221.225303] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2224.406031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2226.618991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2228.674173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2230.747900] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2233.098485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2233.119327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2233.197135] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2233.210088] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2233.245665] Lustre: lustre-MDT0000: new disk, initializing [ 2233.272973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2233.282507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2234.661495] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2238.891257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2238.914351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2238.949983] Lustre: 157616:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2238.977238] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2238.980046] Lustre: Skipped 1 previous similar message [ 2239.020028] Lustre: lustre-MDT0001: new disk, initializing [ 2239.053764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2239.066254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2239.069918] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2240.361981] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2243.486566] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2244.798767] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2247.158776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2247.179218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2247.269614] Lustre: lustre-OST0000: new disk, initializing [ 2247.271765] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2247.301317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2247.379118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2247.382349] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2247.423049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2249.203552] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2252.388639] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2254.437872] Lustre: Failing over lustre-OST0000 [ 2254.468861] LustreError: 160035:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2254.471633] LustreError: 160035:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2254.551318] Lustre: server umount lustre-OST0000 complete [ 2267.428948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2267.535848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2267.541758] Lustre: Skipped 1 previous similar message [ 2267.544695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2267.551678] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2268.961545] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2269.484501] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2270.805015] Lustre: Failing over lustre-MDT0000 [ 2270.868994] LustreError: 161000:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2270.871134] LustreError: 161000:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2270.897448] Lustre: server umount lustre-MDT0000 complete [ 2272.737919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.743044] Lustre: Skipped 2 previous similar messages [ 2273.760138] Lustre: 149548:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1755697680/real 1755697680] req@00000000e828b866 x1840982346751616/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1755697696 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2273.768377] Lustre: 149548:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2277.857116] LustreError: 157625:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2277.863359] LustreError: 157625:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 7 previous similar messages [ 2278.880408] Lustre: 149546:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1755697685/real 1755697685] req@00000000519eacf3 x1840982346752896/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1755697701 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2279.191566] LustreError: 159946:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2282.977273] LustreError: 157625:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2282.982053] LustreError: 157625:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2283.776722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2283.817789] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2283.823254] Lustre: 157618:0:(mgc_request.c:1876:mgc_process_log()) MGC192.168.201.112@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2283.893987] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2283.937414] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2284.311708] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2285.250757] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2289.124597] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2289.131233] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2289.134551] Lustre: Skipped 1 previous similar message [ 2289.148533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2294.245021] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2294.245994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2294.247383] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2294.252942] Lustre: Skipped 1 previous similar message [ 2296.170323] LustreError: 161877:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2296.172288] LustreError: 161877:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2296.233317] Lustre: server umount lustre-OST0000 complete [ 2299.362271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2299.363696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2299.365443] LustreError: Skipped 1 previous similar message [ 2299.370181] Lustre: Skipped 2 previous similar messages [ 2303.640370] Lustre: server umount lustre-MDT0000 complete [ 2304.480959] LustreError: 157625:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2304.488644] LustreError: 157625:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2304.832216] LustreError: 159356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697727 with bad export cookie 10541079713859510523 [ 2304.835050] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2304.836450] LustreError: 159356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2304.984177] Lustre: server umount lustre-MDT0001 complete [ 2308.720692] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 2309.825517] Key type lgssc unregistered [ 2309.955447] LNet: 162809:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2309.957940] LNetError: 162809:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2309.967374] LNet: Removed LNI 192.168.201.112@tcp [ 2310.291849] Key type .llcrypt unregistered [ 2310.293305] Key type ._llcrypt unregistered [ 2330.244986] Key type ._llcrypt registered [ 2330.246438] Key type .llcrypt registered [ 2330.287096] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2330.676235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2330.727458] alg: No test for adler32 (adler32-zlib) [ 2331.575791] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 2331.655109] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 2333.232194] Key type lgssc registered [ 2333.587952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2336.056697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2336.063948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2337.222208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2338.464755] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2341.523405] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2343.441109] LustreError: 164329:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2343.443633] LustreError: 164329:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2343.446037] LustreError: 164329:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 2343.511931] LustreError: 164846:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2343.601184] Lustre: server umount lustre-MDT0000 complete [ 2348.091304] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 2350.985725] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2354.407289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2356.592450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2358.528517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2360.539388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2362.728222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2362.752160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2362.839837] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2362.851883] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2362.884591] Lustre: lustre-MDT0000: new disk, initializing [ 2362.908721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2362.915195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2364.145969] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2368.158434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2368.180312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2368.213238] Lustre: 167473:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2368.233607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2368.235300] Lustre: Skipped 1 previous similar message [ 2368.266459] Lustre: lustre-MDT0001: new disk, initializing [ 2368.298709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2368.309511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2368.313983] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2369.530860] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2372.546947] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2373.691689] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2375.892724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2375.918973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2376.051154] Lustre: lustre-OST0000: new disk, initializing [ 2376.052845] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2376.085620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2377.919389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2377.923077] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2377.967155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2378.024296] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2381.028711] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2382.801744] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2383.320872] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:50:04 (1755697804) [ 2439.648516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2439.648738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2439.651628] Lustre: Skipped 2 previous similar messages [ 2439.653269] Lustre: Skipped 1 previous similar message [ 2442.873571] LustreError: 170953:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2442.875356] LustreError: 170953:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2442.918750] Lustre: server umount lustre-MDT0000 complete [ 2444.084630] LustreError: 169209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697866 with bad export cookie 8897854815956378195 [ 2444.085627] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2444.087775] LustreError: 169209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2444.768428] LustreError: 168200:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2444.768452] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2444.775517] LustreError: 168200:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2444.779108] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2449.888553] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2449.888553] LustreError: 167481:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2450.234952] LustreError: 171154:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2450.237928] LustreError: 171154:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2450.308349] Lustre: server umount lustre-MDT0001 complete [ 2457.642143] LustreError: 171355:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2457.644740] LustreError: 171355:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2457.712625] Lustre: server umount lustre-OST0000 complete [ 2460.513096] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 2463.796314] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2467.422658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2469.702626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2471.852636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2474.153594] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2476.623565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2476.652762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2476.744874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2476.761449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2476.803589] Lustre: lustre-MDT0000: new disk, initializing [ 2476.831928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2476.841431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2478.330791] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2482.495731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2482.519223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2482.551570] Lustre: 173679:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2482.585704] LustreError: 173690:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2482.588887] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2482.593086] Lustre: Skipped 1 previous similar message [ 2482.630736] Lustre: lustre-MDT0001: new disk, initializing [ 2482.678526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2482.682607] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2483.957402] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2487.007963] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2488.198336] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2490.349752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2490.368938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2490.486261] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2490.488247] Lustre: Skipped 1 previous similar message [ 2490.683538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2490.687939] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2490.716339] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2492.202648] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2495.200406] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2496.451336] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2496.526987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2497.697469] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2497.764067] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2501.090564] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2501.091291] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2501.094984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2501.096496] LustreError: Skipped 1 previous similar message [ 2501.099242] Lustre: Skipped 1 previous similar message [ 2504.428200] LustreError: 176588:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2504.430657] LustreError: 176588:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2504.490926] Lustre: server umount lustre-OST0000 complete [ 2506.210900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2506.211828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2506.214370] Lustre: Skipped 2 previous similar messages [ 2506.217549] Lustre: Skipped 1 previous similar message [ 2511.773603] Lustre: server umount lustre-MDT0000 complete [ 2513.079439] LustreError: 174451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697935 with bad export cookie 8897854815956537011 [ 2513.083128] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2513.084359] LustreError: 174451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2513.155251] LustreError: 176993:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2513.158715] LustreError: 176993:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2513.252205] Lustre: server umount lustre-MDT0001 complete [ 2516.206240] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:52:16 (1755697936) [ 2521.551650] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 2524.578724] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2527.909670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2530.156874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2532.255800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2534.412374] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2536.826687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2536.852352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2536.954325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2536.967165] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2536.970075] Lustre: Skipped 1 previous similar message [ 2537.018472] Lustre: lustre-MDT0000: new disk, initializing [ 2537.019871] Lustre: Skipped 1 previous similar message [ 2537.047254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2537.053377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2538.510477] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2542.900087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2542.925341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2542.962641] Lustre: 179967:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2543.082278] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2544.472230] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2547.630453] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2548.904252] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2551.328895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2551.352957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2553.634567] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2556.868096] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2559.993623] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2560.034384] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2565.091491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2565.094563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2565.098859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2565.101129] Lustre: Skipped 3 previous similar messages [ 2566.191149] LustreError: 182494:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2566.193064] LustreError: 182494:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2566.267755] Lustre: server umount lustre-OST0000 complete [ 2568.673155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2568.677085] LustreError: Skipped 1 previous similar message [ 2569.806413] Lustre: server umount lustre-MDT0000 complete [ 2570.209609] LustreError: 180749:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.216167] LustreError: 180749:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2571.086584] LustreError: 179958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755697993 with bad export cookie 8897854815956537865 [ 2571.091068] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2571.093126] LustreError: 179958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2571.264731] Lustre: server umount lustre-MDT0001 complete [ 2574.128387] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:53:14 (1755697994) [ 2579.556614] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 2582.735219] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2586.388540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2588.765969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2590.911739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2593.172222] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2595.589691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2595.608046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2595.692045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2595.702773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2595.704649] Lustre: Skipped 3 previous similar messages [ 2595.737795] Lustre: lustre-MDT0000: new disk, initializing [ 2595.739323] Lustre: Skipped 2 previous similar messages [ 2595.762864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2595.764852] Lustre: Skipped 2 previous similar messages [ 2595.770683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2595.773972] Lustre: Skipped 2 previous similar messages [ 2597.170794] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2601.491448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2601.515712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2601.550232] Lustre: 185871:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2601.666753] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2603.022281] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2606.231970] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2607.551972] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2609.880879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2609.907222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2612.052068] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2614.815701] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2615.192552] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2619.876110] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2619.876477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2619.878905] LustreError: Skipped 1 previous similar message [ 2619.879490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2619.884182] Lustre: Skipped 4 previous similar messages [ 2619.891786] Lustre: Skipped 2 previous similar messages [ 2623.276621] LustreError: 188400:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2623.279022] LustreError: 188400:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2623.356048] Lustre: server umount lustre-OST0000 complete [ 2630.748988] Lustre: server umount lustre-MDT0000 complete [ 2632.121457] LustreError: 186649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698054 with bad export cookie 8897854815956538936 [ 2632.126176] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2632.127823] LustreError: 186649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2632.271594] Lustre: server umount lustre-MDT0001 complete [ 2636.040516] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 2637.208062] Key type lgssc unregistered [ 2637.354478] LNet: 189329:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2637.357594] LNetError: 189329:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2637.366517] LNet: Removed LNI 192.168.201.112@tcp [ 2637.722164] Key type .llcrypt unregistered [ 2637.723186] Key type ._llcrypt unregistered [ 2646.923668] Key type ._llcrypt registered [ 2646.925305] Key type .llcrypt registered [ 2646.969709] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2647.350717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2647.358361] alg: No test for adler32 (adler32-zlib) [ 2648.221488] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 2648.313301] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 2649.904242] Key type lgssc registered [ 2650.317187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2653.068641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2653.076825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2654.255485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2655.604641] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2658.649611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2658.808937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2660.189717] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2662.126046] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2663.434429] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2665.887854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2666.026070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2667.919357] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2669.803249] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2679.333294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2694.626735] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2694.628169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2694.630913] LustreError: Skipped 1 previous similar message [ 2694.637114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2694.641383] Lustre: Skipped 1 previous similar message [ 2697.647106] LustreError: 192948:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2697.711540] Lustre: server umount lustre-OST0000 complete [ 2699.747237] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2699.747312] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2699.750189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2699.754068] Lustre: Skipped 2 previous similar messages [ 2704.865908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2704.868041] Lustre: Skipped 2 previous similar messages [ 2705.089576] LustreError: 193150:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2705.093335] LustreError: 193150:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2705.125346] Lustre: server umount lustre-MDT0000 complete [ 2706.378378] LustreError: 190675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698128 with bad export cookie 1575436836851756090 [ 2706.381776] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2706.385455] LustreError: 190675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2706.443318] LustreError: 193351:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2706.445166] LustreError: 193351:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2706.536571] Lustre: server umount lustre-MDT0001 complete [ 2710.341238] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 2711.462869] Key type lgssc unregistered [ 2711.599429] LNet: 193876:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2711.602187] LNetError: 193876:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2711.611330] LNet: Removed LNI 192.168.201.112@tcp [ 2711.919053] Key type .llcrypt unregistered [ 2711.921042] Key type ._llcrypt unregistered [ 2721.341455] Key type ._llcrypt registered [ 2721.343198] Key type .llcrypt registered [ 2721.382320] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2721.732262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2721.760185] alg: No test for adler32 (adler32-zlib) [ 2722.637728] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 2722.746071] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 2724.344187] Key type lgssc registered [ 2724.784244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2727.686082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2727.694162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2728.883679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2730.259392] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2733.336121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2733.474197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2734.915119] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2736.879345] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2738.105373] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2740.443351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2740.574542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2742.497346] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2744.317524] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2753.895914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2758.113113] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2758.117420] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2758.122506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2759.139241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2759.140146] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2759.142678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2763.117108] LustreError: 197495:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2763.196218] Lustre: server umount lustre-OST0000 complete [ 2764.991780] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffffa01745c45800.ost_server_uuid 50 [ 2765.485041] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffffa01745c45800.ost_server_uuid in DISCONN state after 0 sec [ 2766.036390] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2776.082503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2776.083110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2776.086510] Lustre: Skipped 1 previous similar message [ 2776.088129] Lustre: Skipped 2 previous similar messages [ 2781.153881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2781.156307] Lustre: Skipped 2 previous similar messages [ 2782.206961] LustreError: 197891:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2782.209638] LustreError: 197891:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2782.242934] Lustre: server umount lustre-MDT0000 complete [ 2783.518173] LustreError: 195270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698206 with bad export cookie 10981846285779958579 [ 2783.521467] LustreError: 195270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2783.521886] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2783.590626] LustreError: 198093:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2783.592586] LustreError: 198093:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2783.687122] Lustre: server umount lustre-MDT0001 complete [ 2786.572670] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 09:56:47 (1755698207) [ 2788.872047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2789.045502] LustreError: 198763:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2789.052933] LustreError: 198763:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2789.072369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2790.457221] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2793.452176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2793.589705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2795.046918] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2796.975901] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2798.219293] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2800.663365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2802.802403] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2804.526544] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2806.793349] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2806.826067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2806.892636] Lustre: lustre-OST0001: new disk, initializing [ 2806.894703] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2806.925289] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2806.928529] Lustre: Skipped 1 previous similar message [ 2808.804215] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2810.883140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2810.886971] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2810.919906] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2811.855292] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2813.794600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2813.797701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2818.796102] LustreError: 201943:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2818.798586] LustreError: 201943:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2818.813914] Lustre: server umount lustre-OST0000 complete [ 2819.047061] LustreError: 201760:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2819.055237] LustreError: 201760:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2820.748963] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2820.817575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2822.151338] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2822.227230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2822.424536] LustreError: 201759:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2827.545358] LustreError: 200345:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2827.554024] LustreError: 200345:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 2834.404261] LustreError: 201759:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2834.404459] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2834.405650] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2834.405656] Lustre: Skipped 2 previous similar messages [ 2834.408487] LustreError: 201759:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2834.411416] Lustre: Skipped 1 previous similar message [ 2838.319092] LustreError: 202632:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2838.321085] LustreError: 202632:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2838.392156] Lustre: server umount lustre-OST0001 complete [ 2840.032760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2844.641804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2844.647923] Lustre: Skipped 1 previous similar message [ 2845.781608] Lustre: server umount lustre-MDT0000 complete [ 2846.988253] LustreError: 198742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698269 with bad export cookie 10981846285779959636 [ 2846.991166] LustreError: 198742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2846.992129] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2847.048815] LustreError: 203034:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2847.050708] LustreError: 203034:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2847.135908] Lustre: server umount lustre-MDT0001 complete [ 2848.965820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2850.217602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2851.456619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2852.712918] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2854.754223] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 09:57:55 (1755698275) [ 2856.882442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2856.974627] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2856.986100] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2857.059860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2858.303632] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2861.114336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2861.156685] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2861.169862] Lustre: 204525:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2862.565034] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2864.342192] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2865.580846] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2867.808093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2867.875971] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2869.789456] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2871.543924] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2871.571531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2873.953783] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2873.989547] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2875.929818] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2877.801355] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2879.460428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2879.465039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2879.468363] Lustre: Skipped 5 previous similar messages [ 2880.525240] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 2884.653117] LustreError: 207685:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2884.656023] LustreError: 207685:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2884.670981] Lustre: server umount lustre-OST0000 complete [ 2885.606957] LustreError: 206140:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2885.614670] LustreError: 206140:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2895.843450] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2895.847472] LustreError: Skipped 1 previous similar message [ 2901.418862] Lustre: server umount lustre-OST0001 complete [ 2908.826589] Lustre: server umount lustre-MDT0000 complete [ 2910.082613] LustreError: 204516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698332 with bad export cookie 10981846285779960798 [ 2910.087608] LustreError: 204516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2910.087716] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2910.252734] Lustre: server umount lustre-MDT0001 complete [ 2912.020329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2913.271954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2914.528175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2915.792694] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2917.901582] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 09:58:58 (1755698338) [ 2922.989844] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 2925.917906] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 2929.293027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2931.450097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2933.380480] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2935.382332] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2937.654458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2937.677189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2937.758227] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2937.770871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2937.815201] Lustre: lustre-MDT0000: new disk, initializing [ 2937.839651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2937.841780] Lustre: Skipped 3 previous similar messages [ 2937.847565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2939.154991] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2943.242991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2943.263314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2943.291481] Lustre: 212102:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2943.324548] LustreError: 212113:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2943.327520] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2943.331152] LustreError: 212113:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 8 previous similar messages [ 2943.333323] Lustre: Skipped 1 previous similar message [ 2943.375914] Lustre: lustre-MDT0001: new disk, initializing [ 2943.421915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2943.424810] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2944.656150] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2947.765029] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2948.984151] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2951.250404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2951.275912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2951.375666] Lustre: lustre-OST0000: new disk, initializing [ 2951.378424] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2953.301294] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2956.455589] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2957.808479] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2959.349126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2959.352055] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2959.383629] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2959.925475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 2961.215677] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2961.293193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2964.450130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2964.450422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2964.452673] LustreError: Skipped 1 previous similar message [ 2964.456226] Lustre: Skipped 5 previous similar messages [ 2964.456846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2964.461280] Lustre: Skipped 10 previous similar messages [ 2968.044064] LustreError: 215024:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2968.046026] LustreError: 215024:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2968.109251] Lustre: server umount lustre-OST0000 complete [ 2974.177078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2975.519569] Lustre: server umount lustre-MDT0000 complete [ 2976.774264] LustreError: 212094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698399 with bad export cookie 10981846285779962534 [ 2976.778213] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2976.778721] LustreError: 212094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2976.921730] Lustre: server umount lustre-MDT0001 complete [ 2979.796763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2979.952799] LustreError: 215904:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2981.361112] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2984.257404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2985.678689] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2987.454820] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2988.586691] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2990.858105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2992.850622] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 2994.682236] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2995.967220] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3007.271607] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 11 sec [ 3008.589644] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3008.665349] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3009.504837] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3015.456032] Lustre: server umount lustre-OST0000 complete [ 3017.261154] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3018.487888] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3042.051844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3042.184223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3042.187277] Lustre: Skipped 5 previous similar messages [ 3044.023648] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3045.808479] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3046.331656] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3049.325089] LustreError: 220322:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3049.328101] LustreError: 220322:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3049.385946] Lustre: server umount lustre-OST0000 complete [ 3050.566205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3050.567551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3050.572163] Lustre: Skipped 6 previous similar messages [ 3050.575160] Lustre: Skipped 6 previous similar messages [ 3052.690562] Lustre: server umount lustre-MDT0000 complete [ 3053.868394] LustreError: 215884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698476 with bad export cookie 10981846285779963416 [ 3053.871662] LustreError: 215884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3053.872226] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3054.016290] Lustre: server umount lustre-MDT0001 complete [ 3056.568018] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:01:17 (1755698477) [ 3058.665126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3058.810700] LustreError: 221398:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3060.087247] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3062.817933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3064.205332] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3065.967221] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3067.158800] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3069.324492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3071.210859] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3072.910694] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3074.249718] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3074.319086] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3075.551163] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3075.625791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3077.814704] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3077.839348] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3077.899246] Lustre: lustre-OST0001: new disk, initializing [ 3077.901147] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3079.862480] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3082.894746] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3084.279343] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3085.826505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3085.831809] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3085.864233] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3086.393361] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3087.656293] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3087.736152] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3090.916843] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3090.920902] LustreError: Skipped 2 previous similar messages [ 3094.462660] Lustre: server umount lustre-OST0001 complete [ 3096.351203] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3096.423654] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3097.699723] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3097.783499] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3121.328737] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3123.160615] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3124.859067] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3125.459753] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3128.183846] Lustre: server umount lustre-OST0001 complete [ 3135.779284] Lustre: server umount lustre-OST0000 complete [ 3143.253936] Lustre: server umount lustre-MDT0000 complete [ 3144.530284] LustreError: 224368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698567 with bad export cookie 10981846285779964347 [ 3144.534213] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3144.534621] LustreError: 224368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3150.738486] Lustre: server umount lustre-MDT0001 complete [ 3152.630595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3153.927947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3155.160830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3156.376887] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3158.387301] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:02:59 (1755698579) [ 3160.477476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3160.580077] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3160.589712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3161.825311] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3164.563633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3164.593251] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3164.602209] Lustre: 229295:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3165.874136] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3167.584292] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3168.721687] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3170.824048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3170.886569] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3170.941032] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3170.942681] Lustre: Skipped 7 previous similar messages [ 3172.674550] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3174.285145] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3183.399152] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3183.423174] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3185.481095] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3187.504813] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3192.309428] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 3193.529507] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3193.598748] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3194.910932] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3194.984415] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3196.113303] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa01758501000.ost_server_uuid 50 [ 3196.614210] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa01758501000.ost_server_uuid in FULL state after 0 sec [ 3196.947500] Lustre: Permanently deactivating lustre-OST0001 [ 3196.954908] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3197.684880] Lustre: Permanently reactivating lustre-OST0001 [ 3200.485341] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3200.485809] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3200.488098] LustreError: Skipped 1 previous similar message [ 3200.492111] Lustre: Skipped 6 previous similar messages [ 3200.496854] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3200.498643] Lustre: Skipped 12 previous similar messages [ 3204.651121] LustreError: 233244:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3204.654266] LustreError: 233244:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 3204.670928] Lustre: server umount lustre-OST0001 complete [ 3205.605029] LustreError: 230908:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3205.613214] LustreError: 230908:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 22 previous similar messages [ 3212.024570] Lustre: server umount lustre-OST0000 complete [ 3219.475926] Lustre: server umount lustre-MDT0000 complete [ 3220.722385] LustreError: 229287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698643 with bad export cookie 10981846285779965607 [ 3220.725968] LustreError: 229287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3220.726281] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3220.868512] Lustre: server umount lustre-MDT0001 complete [ 3222.718495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3223.981191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3225.299949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3226.573079] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3228.639091] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:04:09 (1755698649) [ 3228.965074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3231.190194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3231.290764] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3231.301886] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3231.306244] Lustre: Skipped 5 previous similar messages [ 3232.582856] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3235.371438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3235.404524] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3235.413780] Lustre: 235400:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3236.771533] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3238.457951] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3239.652546] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3242.165276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3242.196517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3242.287814] Lustre: Permanently deactivating lustre-OST0000 [ 3242.293782] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3243.184556] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3244.333951] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3246.080965] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3248.285033] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3250.156444] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3252.019744] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3253.326180] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3258.017110] Lustre: Permanently reactivating lustre-OST0000 [ 3259.321357] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3259.346335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3271.938931] Lustre: server umount lustre-OST0001 complete [ 3279.715145] Lustre: server umount lustre-OST0000 complete [ 3287.191036] Lustre: server umount lustre-MDT0000 complete [ 3288.445688] LustreError: 235392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698710 with bad export cookie 10981846285779967245 [ 3288.450330] LustreError: 235392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3288.451399] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3288.598652] Lustre: server umount lustre-MDT0001 complete [ 3292.236653] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 3293.473964] Key type lgssc unregistered [ 3293.614524] LNet: 239762:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3293.618477] LNetError: 239762:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3293.627549] LNet: Removed LNI 192.168.201.112@tcp [ 3293.980565] Key type .llcrypt unregistered [ 3293.981912] Key type ._llcrypt unregistered [ 3302.676045] Key type ._llcrypt registered [ 3302.677032] Key type .llcrypt registered [ 3302.709721] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3303.055510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3303.141508] alg: No test for adler32 (adler32-zlib) [ 3303.998654] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 3304.086883] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 3305.664146] Key type lgssc registered [ 3306.017358] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3307.194728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3309.385407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3309.391370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3310.545575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3311.784622] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3314.746069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3314.772862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3314.803917] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3314.815391] Lustre: 241170:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3314.819101] Lustre: Permanently deactivating lustre-MDT0001 [ 3314.824348] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3314.951422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3315.694880] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3315.698625] LustreError: 241212:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3315.702188] LustreError: 241212:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3315.704810] LustreError: 241212:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 5, retries 1, failed: rc = -108 [ 3316.338566] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3318.118320] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3319.301664] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3321.566715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3321.685128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3323.547685] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3324.774167] LustreError: 242788:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3324.778397] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3324.780065] LustreError: 242788:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3325.444147] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3327.886400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3327.961620] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3329.975671] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3330.292905] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3331.908505] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3336.424555] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3336.425041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3337.921470] Lustre: Permanently reactivating lustre-MDT0001 [ 3337.926886] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3337.928992] Lustre: Skipped 2 previous similar messages [ 3341.982424] Lustre: lustre-MDT0001: Client c9939091-a58b-4a08-ac87-45df211d305f (at 192.168.201.12@tcp) reconnecting [ 3343.223642] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3343.739423] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3345.506660] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3345.580638] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3346.872206] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3346.946784] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3347.902849] Lustre: Permanently deactivating lustre-MDT0001 [ 3347.912421] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3347.919166] Lustre: Skipped 2 previous similar messages [ 3353.599325] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3361.249843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3361.253541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3361.259426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3362.274249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3362.277962] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3362.281229] Lustre: Skipped 2 previous similar messages [ 3365.255921] LustreError: 245259:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3365.298420] Lustre: server umount lustre-MDT0000 complete [ 3366.709468] LustreError: 241899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698789 with bad export cookie 14044753388254312682 [ 3366.712418] LustreError: 241899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3366.712959] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3366.783396] LustreError: 245461:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3366.787402] LustreError: 245461:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3366.878721] Lustre: server umount lustre-MDT0001 complete [ 3374.323108] LustreError: 245661:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3374.327471] LustreError: 245661:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3374.349264] Lustre: server umount lustre-OST0000 complete [ 3378.407084] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:06:39 (1755698799) [ 3381.817554] LustreError: 246666:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3381.820202] LustreError: 246666:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3381.886076] Lustre: server umount lustre-OST0001 complete [ 3383.815888] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 3386.683715] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3389.828169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3391.899265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3393.878986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3395.819491] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3398.099840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3398.125732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3398.221626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3398.224613] Lustre: Skipped 2 previous similar messages [ 3398.235062] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3398.274621] Lustre: lustre-MDT0000: new disk, initializing [ 3398.301608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3398.309112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3399.666260] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3403.642730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3403.669598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3403.706434] Lustre: 248834:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3403.730656] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3403.732476] Lustre: Skipped 1 previous similar message [ 3403.768332] Lustre: lustre-MDT0001: new disk, initializing [ 3403.808738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3403.811749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3404.993861] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3407.985601] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3409.076321] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3411.277664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3411.302852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3411.391283] Lustre: lustre-OST0000: new disk, initializing [ 3411.393164] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3411.421116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3411.423045] Lustre: Skipped 1 previous similar message [ 3413.200728] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3415.034299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3415.037332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3415.069450] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3416.167160] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3416.778165] LustreError: 248838:0:(mdt_handler.c:3249:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3421.325445] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3421.355389] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3421.435877] Lustre: lustre-OST0001: new disk, initializing [ 3421.438123] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3422.528668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3422.533159] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3422.566994] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3423.421340] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3426.651647] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3426.872073] LustreError: 248838:0:(mdt_handler.c:3249:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3426.880511] LustreError: 248838:0:(mdt_handler.c:3249:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3427.179563] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3436.968134] LustreError: 248838:0:(mdt_handler.c:3249:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3438.054445] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3438.054814] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3438.057714] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3438.062988] Lustre: Skipped 1 previous similar message [ 3438.064682] Lustre: Skipped 3 previous similar messages [ 3440.927703] Lustre: lustre-OST0001: Not available for connect from 192.168.201.12@tcp (stopping) [ 3440.929445] Lustre: Skipped 1 previous similar message [ 3443.629079] LustreError: 252306:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3443.631064] LustreError: 252306:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3443.645977] Lustre: server umount lustre-OST0001 complete [ 3445.729616] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3445.734718] LustreError: Skipped 1 previous similar message [ 3445.737814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3445.745511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3445.748462] Lustre: Skipped 3 previous similar messages [ 3448.292435] LustreError: 250551:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3448.297532] LustreError: 250551:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3451.253618] Lustre: server umount lustre-OST0000 complete [ 3453.409879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3453.414678] Lustre: Skipped 2 previous similar messages [ 3458.529097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3458.531145] Lustre: Skipped 5 previous similar messages [ 3458.759330] LustreError: 252710:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3458.762941] LustreError: 252710:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3458.797984] Lustre: server umount lustre-MDT0000 complete [ 3460.122347] LustreError: 248824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698882 with bad export cookie 14044753388254315489 [ 3460.126823] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3460.129445] LustreError: 248824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3466.331777] Lustre: server umount lustre-MDT0001 complete [ 3469.868127] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 3471.225403] Key type lgssc unregistered [ 3471.361415] LNet: 253438:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3471.364578] LNetError: 253438:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3471.374336] LNet: Removed LNI 192.168.201.112@tcp [ 3471.666677] Key type .llcrypt unregistered [ 3471.668265] Key type ._llcrypt unregistered [ 3484.955602] Key type ._llcrypt registered [ 3484.957372] Key type .llcrypt registered [ 3485.029842] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 3492.211211] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3492.723754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3492.740900] alg: No test for adler32 (adler32-zlib) [ 3493.625091] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 3493.719991] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 3495.304275] Key type lgssc registered [ 3495.833148] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3499.298230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3501.983517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3504.416225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3507.027377] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3509.365210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3509.389182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3509.396145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3510.501169] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3510.517816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3510.562560] Lustre: lustre-MDT0000: new disk, initializing [ 3510.596107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3510.602908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3512.078722] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3516.380868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3516.406145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3516.455383] Lustre: 257348:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3516.483909] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3516.485678] Lustre: Skipped 1 previous similar message [ 3516.530640] Lustre: lustre-MDT0001: new disk, initializing [ 3516.578155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3516.595503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3516.600179] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3517.972733] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3521.276784] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3522.436635] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3524.645865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3524.675099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3524.782462] Lustre: lustre-OST0000: new disk, initializing [ 3524.785187] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3524.820880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3526.825531] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3530.004637] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3531.583088] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3531.780329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3531.784206] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3531.829864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3532.688549] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3534.080546] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3534.171802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3536.870431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3536.871262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3536.874320] LustreError: Skipped 1 previous similar message [ 3536.878119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3536.881885] Lustre: Skipped 1 previous similar message [ 3540.910141] LustreError: 260264:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3540.986365] Lustre: server umount lustre-OST0000 complete [ 3542.496903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3542.499569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3542.502866] Lustre: Skipped 1 previous similar message [ 3542.504419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3547.105512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3547.105918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3548.348656] LustreError: 260465:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3548.350456] LustreError: 260465:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3548.376708] Lustre: server umount lustre-MDT0000 complete [ 3549.730507] LustreError: 259081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755698972 with bad export cookie 17728738775958739186 [ 3549.734850] LustreError: 259081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3549.737174] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3549.803983] LustreError: 260667:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3549.805739] LustreError: 260667:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3549.906129] Lustre: server umount lustre-MDT0001 complete [ 3552.898695] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 10:09:33 (1755698973) [ 3555.212900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3555.414953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3556.763823] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3559.618039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3559.752059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3561.082090] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3563.134536] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3564.303391] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3566.695581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3568.692979] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3570.722410] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3621.346854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3621.349134] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3621.350522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3621.352234] Lustre: Skipped 1 previous similar message [ 3621.354414] Lustre: Skipped 1 previous similar message [ 3625.388216] LustreError: 263661:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3625.390959] LustreError: 263661:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3625.465305] Lustre: server umount lustre-OST0000 complete [ 3626.879850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3631.026824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3631.164738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3631.166615] Lustre: Skipped 1 previous similar message [ 3633.170356] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3635.156322] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3641.442156] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3641.445460] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3641.445580] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.201.112@tcp (at 0@lo) [ 3641.452216] Lustre: Skipped 1 previous similar message [ 3672.548107] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3672.548817] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3672.550905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3672.550911] Lustre: Skipped 1 previous similar message [ 3672.553982] Lustre: Skipped 1 previous similar message [ 3672.565232] LustreError: Skipped 2 previous similar messages [ 3675.247147] LustreError: 265318:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3675.250531] LustreError: 265318:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3675.354888] Lustre: server umount lustre-OST0000 complete [ 3677.667200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3682.786536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3682.790485] Lustre: Skipped 4 previous similar messages [ 3683.123790] Lustre: server umount lustre-MDT0000 complete [ 3684.460059] LustreError: 261361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699106 with bad export cookie 17728738775958740068 [ 3684.462825] LustreError: 261361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3684.463822] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3684.523812] LustreError: 265721:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3684.526188] LustreError: 265721:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3684.612654] Lustre: server umount lustre-MDT0001 complete [ 3688.414946] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 3689.615454] Key type lgssc unregistered [ 3689.759449] LNet: 266247:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3689.762739] LNetError: 266247:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3689.770439] LNet: Removed LNI 192.168.201.112@tcp [ 3690.104972] Key type .llcrypt unregistered [ 3690.106349] Key type ._llcrypt unregistered [ 3700.049139] Key type ._llcrypt registered [ 3700.050377] Key type .llcrypt registered [ 3700.083123] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3700.375631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3700.403127] alg: No test for adler32 (adler32-zlib) [ 3701.254531] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 3701.342785] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 3702.928172] Key type lgssc registered [ 3703.393758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3706.529887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3706.542673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3707.727304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3709.294832] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3713.196075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3713.329224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3714.803424] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3717.244463] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3718.867763] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3722.158985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3722.360101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3724.687937] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3726.923369] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3732.587799] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3748.324484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3748.325749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3748.326413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3748.332130] Lustre: Skipped 1 previous similar message [ 3748.979349] LustreError: 270651:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3749.098086] Lustre: server umount lustre-OST0000 complete [ 3753.443265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3753.444568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3753.451859] Lustre: Skipped 1 previous similar message [ 3753.456153] Lustre: Skipped 1 previous similar message [ 3758.561843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3758.565031] Lustre: Skipped 2 previous similar messages [ 3763.681096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3763.684023] Lustre: Skipped 1 previous similar message [ 3765.216126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3765.273234] LustreError: 270852:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3765.277168] LustreError: 270852:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3765.312961] Lustre: server umount lustre-MDT0000 complete [ 3767.108866] LustreError: 267609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699189 with bad export cookie 3658393526842623914 [ 3767.116166] LustreError: 267609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3767.116555] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3767.202987] LustreError: 271056:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3767.207563] LustreError: 271056:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3767.309128] Lustre: server umount lustre-MDT0001 complete [ 3772.280938] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 3773.688579] Key type lgssc unregistered [ 3773.874759] LNet: 271583:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3773.880235] LNetError: 271583:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3773.892615] LNet: Removed LNI 192.168.201.112@tcp [ 3774.359130] Key type .llcrypt unregistered [ 3774.361415] Key type ._llcrypt unregistered [ 3781.998283] Key type ._llcrypt registered [ 3782.000413] Key type .llcrypt registered [ 3782.041709] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3782.328474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3782.335103] alg: No test for adler32 (adler32-zlib) [ 3783.185361] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 3783.269936] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 3784.856127] Key type lgssc registered [ 3785.239428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3787.812793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3787.819456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3788.971173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3790.195677] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3793.163199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3793.285063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3794.571905] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3796.567103] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3798.205505] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3800.648879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3800.842713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3802.887527] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3804.647874] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3815.207280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3825.635257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3825.635623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3825.638993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3825.642836] Lustre: Skipped 1 previous similar message [ 3825.646783] Lustre: Skipped 1 previous similar message [ 3829.869539] LustreError: 275154:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3829.953522] Lustre: server umount lustre-OST0000 complete [ 3834.337570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3834.340884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3834.345910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3835.874872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3835.876279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3837.442204] LustreError: 275356:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3837.444784] LustreError: 275356:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3837.470278] Lustre: server umount lustre-MDT0000 complete [ 3838.673158] LustreError: 272737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699261 with bad export cookie 17248918647136717251 [ 3838.677112] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3838.678325] LustreError: 272737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3838.742413] LustreError: 275558:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3838.745207] LustreError: 275558:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3838.819278] Lustre: server umount lustre-MDT0001 complete [ 3842.793371] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 3843.868099] Key type lgssc unregistered [ 3844.002386] LNet: 276083:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3844.005602] LNetError: 276083:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3844.012268] LNet: Removed LNI 192.168.201.112@tcp [ 3844.314796] Key type .llcrypt unregistered [ 3844.315682] Key type ._llcrypt unregistered [ 3852.773133] Key type ._llcrypt registered [ 3852.774174] Key type .llcrypt registered [ 3852.811522] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3853.099072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3853.104766] alg: No test for adler32 (adler32-zlib) [ 3853.950580] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 3854.026863] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 3855.600169] Key type lgssc registered [ 3856.081650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3859.049218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3859.057799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3860.223710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3861.677430] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3865.039939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3865.208548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3866.601820] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3868.635669] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3870.223868] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3873.201853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3873.336329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3875.256076] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3877.193246] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3884.583058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3899.876397] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3899.877130] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3899.878547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3899.879466] Lustre: Skipped 1 previous similar message [ 3902.831161] LustreError: 280471:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3902.899412] Lustre: server umount lustre-OST0000 complete [ 3904.993335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3904.993976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3904.997777] Lustre: Skipped 1 previous similar message [ 3904.999943] Lustre: Skipped 1 previous similar message [ 3910.113815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3910.116341] Lustre: Skipped 2 previous similar messages [ 3910.267783] LustreError: 280673:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3910.269864] LustreError: 280673:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3910.295592] Lustre: server umount lustre-MDT0000 complete [ 3911.495956] LustreError: 280674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699334 with bad export cookie 10982234286626264710 [ 3911.499185] LustreError: 280674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3911.499353] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3911.566602] LustreError: 280875:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3911.570012] LustreError: 280875:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3911.656183] Lustre: server umount lustre-MDT0001 complete [ 3915.152922] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 3916.207761] Key type lgssc unregistered [ 3916.348528] LNet: 281400:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3916.350882] LNetError: 281400:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3916.357421] LNet: Removed LNI 192.168.201.112@tcp [ 3916.684739] Key type .llcrypt unregistered [ 3916.685880] Key type ._llcrypt unregistered [ 3923.824730] Key type ._llcrypt registered [ 3923.826634] Key type .llcrypt registered [ 3923.866612] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3924.346901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3924.368745] alg: No test for adler32 (adler32-zlib) [ 3925.208453] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 3925.296085] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 3926.872148] Key type lgssc registered [ 3927.210970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3929.567017] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3929.574506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3930.743761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3932.044533] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3934.837553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3934.952956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3936.167276] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3937.793200] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3938.912654] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3940.996532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3941.099531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3942.119552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3942.798511] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 3944.407278] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3950.567844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3950.567882] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3950.569431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3950.571200] Lustre: Skipped 1 previous similar message [ 3954.862518] LustreError: 284980:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3954.938201] Lustre: server umount lustre-OST0000 complete [ 3960.801890] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3960.801920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3960.802638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3960.802643] Lustre: Skipped 1 previous similar message [ 3960.811912] Lustre: Skipped 1 previous similar message [ 3962.556914] LustreError: 285183:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3962.558765] LustreError: 285183:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3962.583947] Lustre: server umount lustre-MDT0000 complete [ 3963.737396] LustreError: 282554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699386 with bad export cookie 1262291333976075580 [ 3963.739859] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3963.740871] LustreError: 282554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3963.804860] LustreError: 285385:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3963.807275] LustreError: 285385:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3963.890809] Lustre: server umount lustre-MDT0001 complete [ 3967.167163] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 3968.154331] Key type lgssc unregistered [ 3968.277297] LNet: 285911:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3968.281508] LNetError: 285911:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3968.289353] LNet: Removed LNI 192.168.201.112@tcp [ 3968.582623] Key type .llcrypt unregistered [ 3968.583517] Key type ._llcrypt unregistered [ 3971.728728] Key type ._llcrypt registered [ 3971.729711] Key type .llcrypt registered [ 3971.764102] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 3976.728883] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 3982.126063] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 3982.507945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3982.525581] alg: No test for adler32 (adler32-zlib) [ 3983.368459] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 3983.452018] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 3985.024121] Key type lgssc registered [ 3985.412272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3988.131057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3990.040692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3991.756566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3993.655865] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3995.765114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3995.783230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3995.788095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3996.863571] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3996.873902] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3996.909661] Lustre: lustre-MDT0000: new disk, initializing [ 3996.932574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3996.938422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3998.086539] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4002.265849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4002.292496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4002.331401] Lustre: 289372:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4002.352425] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4002.354670] Lustre: Skipped 1 previous similar message [ 4002.388235] Lustre: lustre-MDT0001: new disk, initializing [ 4002.419698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4002.430178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4002.432907] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4003.628315] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4006.531702] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4007.581624] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4009.732689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4009.755427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4009.851353] Lustre: lustre-OST0000: new disk, initializing [ 4009.853608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4009.880371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4011.193678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4011.196223] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4011.233527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4011.591322] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4014.496312] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4015.707137] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4015.779841] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4016.993258] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4017.072111] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4021.730168] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4021.731933] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4021.732125] LustreError: Skipped 1 previous similar message [ 4021.732716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4021.734958] Lustre: Skipped 1 previous similar message [ 4021.740524] Lustre: Skipped 1 previous similar message [ 4026.850266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4026.852765] Lustre: Skipped 1 previous similar message [ 4031.968133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4031.971674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4031.973152] Lustre: Skipped 1 previous similar message [ 4031.990566] LustreError: 292280:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4032.049802] Lustre: server umount lustre-OST0000 complete [ 4033.504865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4033.508035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4037.089219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4037.089893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4037.090783] Lustre: Skipped 1 previous similar message [ 4039.290635] LustreError: 292483:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4039.292308] LustreError: 292483:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4039.318413] Lustre: server umount lustre-MDT0000 complete [ 4040.426085] LustreError: 289363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699462 with bad export cookie 69722837122571143 [ 4040.428731] LustreError: 289363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4040.429303] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4040.489835] LustreError: 292685:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4040.492308] LustreError: 292685:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4040.569742] Lustre: server umount lustre-MDT0001 complete [ 4043.028731] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:17:43 (1755699463) [ 4044.962571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4045.130202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4046.246522] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4048.850534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4050.212197] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4052.010163] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4053.100131] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4055.411068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4055.538604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4055.541267] Lustre: Skipped 1 previous similar message [ 4057.413267] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4059.307794] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4073.952919] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4073.955086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4073.959540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4073.961522] Lustre: Skipped 1 previous similar message [ 4079.404072] LustreError: 295601:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4079.405956] LustreError: 295601:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4079.464943] Lustre: server umount lustre-OST0000 complete [ 4084.194269] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 4084.196632] LustreError: Skipped 1 previous similar message [ 4084.198027] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4084.201570] Lustre: Skipped 1 previous similar message [ 4084.203242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4084.204968] Lustre: Skipped 2 previous similar messages [ 4086.779972] LustreError: 295804:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4086.781726] LustreError: 295804:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4086.808237] Lustre: server umount lustre-MDT0000 complete [ 4087.889173] LustreError: 295806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699510 with bad export cookie 69722837122572011 [ 4087.892474] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4087.894740] LustreError: 295806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4088.035874] Lustre: server umount lustre-MDT0001 complete [ 4091.230284] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 4092.239955] Key type lgssc unregistered [ 4092.370397] LNet: 296533:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4092.372617] LNetError: 296533:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4092.382411] LNet: Removed LNI 192.168.201.112@tcp [ 4092.695979] Key type .llcrypt unregistered [ 4092.697195] Key type ._llcrypt unregistered [ 4100.626470] Key type ._llcrypt registered [ 4100.627526] Key type .llcrypt registered [ 4100.660808] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 4106.766510] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 4107.131439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4107.142860] alg: No test for adler32 (adler32-zlib) [ 4107.982506] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 4108.053697] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 4109.624133] Key type lgssc registered [ 4109.938722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4112.650173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4114.860291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4116.857344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4118.946889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4120.751309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4123.069893] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4126.125846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4126.146581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4126.151712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4127.243381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4127.257621] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4127.302582] Lustre: lustre-MDT0000: new disk, initializing [ 4127.330151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4127.342185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4128.856954] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4134.091578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4134.121901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4134.163520] Lustre: 300297:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4134.203828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4134.206943] Lustre: Skipped 1 previous similar message [ 4134.250453] Lustre: lustre-MDT0001: new disk, initializing [ 4134.294563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4134.314168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4134.318926] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4136.096953] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4140.004194] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4141.287458] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4144.358051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4144.396925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4144.553912] Lustre: lustre-OST2710: new disk, initializing [ 4144.556699] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4144.597488] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4144.849135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4144.855110] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4144.914042] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4147.163692] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4150.936283] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4153.856133] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4153.887780] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4153.948111] Lustre: lustre-OST03e8: new disk, initializing [ 4153.950415] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4153.994001] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4155.910355] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4157.808832] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4160.602020] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4160.690300] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4162.263712] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4162.356687] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4162.567987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4162.572204] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4162.607904] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4163.895255] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4163.984261] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4165.638987] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4165.715673] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4167.652142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4167.653532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4167.656854] Lustre: Skipped 1 previous similar message [ 4167.660612] Lustre: Skipped 3 previous similar messages [ 4172.770330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4172.775392] Lustre: Skipped 3 previous similar messages [ 4172.993860] LustreError: 304590:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4173.028900] Lustre: server umount lustre-MDT0000 complete [ 4174.631746] LustreError: 304504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699597 with bad export cookie 12750839134922221056 [ 4174.636717] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4174.641369] LustreError: 304504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4174.735257] LustreError: 304791:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4174.738329] LustreError: 304791:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4174.835147] Lustre: server umount lustre-MDT0001 complete [ 4186.104205] LustreError: 304992:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4186.109772] LustreError: 304992:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4186.139236] Lustre: server umount lustre-OST2710 complete [ 4196.661032] LustreError: 305194:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4196.665491] LustreError: 305194:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4196.741737] Lustre: server umount lustre-OST03e8 complete [ 4203.609754] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 4207.185433] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 4210.617502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4213.505608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4215.689913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4218.362181] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4220.996588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4221.037373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4221.159741] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4221.177186] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4221.230499] Lustre: lustre-MDT0000: new disk, initializing [ 4221.259024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4221.265707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4222.603865] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4227.462646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4227.495483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4227.533079] Lustre: 307973:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4227.647744] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4229.043568] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4231.997782] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4233.081733] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4235.171729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4235.195077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4235.285803] Lustre: lustre-OST0000: new disk, initializing [ 4235.287189] Lustre: Skipped 1 previous similar message [ 4235.289026] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4235.291384] Lustre: Skipped 2 previous similar messages [ 4235.316278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4235.317895] Lustre: Skipped 1 previous similar message [ 4237.061589] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4239.945978] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4240.374738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4240.376726] Lustre: Skipped 1 previous similar message [ 4240.378306] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4240.407848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4241.141258] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4241.210711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4242.478253] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4242.567449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4245.474028] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4245.475927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4245.476177] LustreError: Skipped 1 previous similar message [ 4245.477479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4245.479295] Lustre: Skipped 3 previous similar messages [ 4249.262929] LustreError: 310883:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4249.264539] LustreError: 310883:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4249.326202] Lustre: server umount lustre-OST0000 complete [ 4250.593284] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4250.597698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4250.599233] Lustre: Skipped 2 previous similar messages [ 4255.713193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4255.715480] Lustre: Skipped 1 previous similar message [ 4256.668738] Lustre: server umount lustre-MDT0000 complete [ 4257.863553] LustreError: 307966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699680 with bad export cookie 12750839134922222729 [ 4257.866921] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4257.868786] LustreError: 307966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4257.932289] LustreError: 311286:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4257.935400] LustreError: 311286:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4258.017420] Lustre: server umount lustre-MDT0001 complete [ 4260.722775] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 10:21:21 (1755699681) [ 4261.176205] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4261.732045] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 10:21:22 (1755699682) [ 4262.685605] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 4265.182394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4266.413584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4267.618096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4268.856072] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4269.221568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4271.304803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4271.403543] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4271.411111] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4271.476196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4272.646477] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4275.243141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4275.274220] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4275.283208] Lustre: 313405:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4276.527632] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4278.163770] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4279.272055] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4281.396445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4281.425160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4281.489302] Lustre: Denying initial registration attempt from nid 192.168.201.112@tcp, specified as failover [ 4281.491917] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4281.496620] LustreError: 315008:0:(tgt_mount.c:2460:server_fill_super()) Unable to start targets: -99 [ 4281.501531] LustreError: 315008:0:(tgt_mount.c:1954:server_put_super()) no obd lustre-OST0000 [ 4281.505676] LustreError: 315008:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4281.554126] Lustre: server umount lustre-OST0000 complete [ 4281.555507] LustreError: 315008:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4285.921898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4285.921981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4285.922622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4285.930290] Lustre: Skipped 2 previous similar messages [ 4289.724866] LustreError: 315376:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4289.727898] LustreError: 315376:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4289.750382] Lustre: server umount lustre-MDT0000 complete [ 4290.890442] LustreError: 313396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699713 with bad export cookie 12750839134922223597 [ 4290.893515] LustreError: 313396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4290.893788] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4291.034738] Lustre: server umount lustre-MDT0001 complete [ 4294.565825] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 4295.551029] Key type lgssc unregistered [ 4295.679330] LNet: 316104:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4295.681388] LNetError: 316104:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4295.688275] LNet: Removed LNI 192.168.201.112@tcp [ 4295.994569] Key type .llcrypt unregistered [ 4295.995789] Key type ._llcrypt unregistered [ 4299.212170] Key type ._llcrypt registered [ 4299.213085] Key type .llcrypt registered [ 4299.245325] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 4299.584666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4299.623727] alg: No test for adler32 (adler32-zlib) [ 4300.465128] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 4300.541169] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 4302.112138] Key type lgssc registered [ 4302.446788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4304.779160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4306.049815] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4307.265836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4308.473163] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4308.798648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4315.184325] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 4317.745843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4317.752272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4318.837177] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4318.844282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4318.902169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4320.056165] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4322.656819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4322.686928] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4322.697770] Lustre: 318822:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4322.787597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4323.926737] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4325.581123] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4326.664768] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4328.696419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4328.720885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4328.782879] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4328.819462] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4328.869772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4330.593109] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4332.202624] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4332.826154] LustreError: 321081:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4332.885814] Lustre: server umount lustre-OST0000 complete [ 4334.050193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4334.051322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4334.055947] Lustre: Skipped 1 previous similar message [ 4339.169083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4339.170718] Lustre: Skipped 2 previous similar messages [ 4344.289060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4344.291080] Lustre: Skipped 1 previous similar message [ 4348.384123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4348.412754] LustreError: 321282:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 4348.415811] LustreError: 321282:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4348.443668] Lustre: server umount lustre-MDT0000 complete [ 4349.670139] LustreError: 318811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699772 with bad export cookie 15179618663811523041 [ 4349.673130] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4349.673785] LustreError: 318811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4349.737171] LustreError: 321487:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 4349.738801] LustreError: 321487:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4349.820066] Lustre: server umount lustre-MDT0001 complete [ 4353.120124] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 4354.074948] Key type lgssc unregistered [ 4354.204329] LNet: 322013:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4354.207045] LNetError: 322013:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4354.215299] LNet: Removed LNI 192.168.201.112@tcp [ 4354.512158] Key type .llcrypt unregistered [ 4354.513450] Key type ._llcrypt unregistered [ 4363.015253] Key type ._llcrypt registered [ 4363.016711] Key type .llcrypt registered [ 4363.053534] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 4363.454792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4363.485085] alg: No test for adler32 (adler32-zlib) [ 4364.362915] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 4364.448551] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 4366.032189] Key type lgssc registered [ 4366.470228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4369.650497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4369.663133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4370.857272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4372.393671] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4375.143308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4375.258054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4376.472338] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4378.093339] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4379.303690] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4381.714573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4381.848434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4383.796242] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4385.674598] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4401.120807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4401.122662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4401.125760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4402.470441] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.12@tcp (stopping) [ 4402.659116] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4402.662373] Lustre: Skipped 1 previous similar message [ 4405.436201] LustreError: 325629:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4405.465709] Lustre: server umount lustre-MDT0000 complete [ 4407.146552] LustreError: 323362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699829 with bad export cookie 9904321582928429392 [ 4407.151482] LustreError: 323362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4407.152153] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4407.576593] LustreError: 325632:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4407.586737] LustreError: 325632:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4407.588607] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.12@tcp (stopping) [ 4407.596806] Lustre: Skipped 3 previous similar messages [ 4407.777656] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4412.695699] LustreError: 325632:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4412.699505] LustreError: 325632:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4412.699942] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.12@tcp (stopping) [ 4412.703808] Lustre: Skipped 1 previous similar message [ 4413.306149] LustreError: 325833:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4413.307806] LustreError: 325833:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4413.380789] Lustre: server umount lustre-MDT0001 complete [ 4415.419144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4418.903688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4432.352768] LustreError: 322691:0:(client.c:1385:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000000f3325e x1840984658828928/t0(0) o250->MGC192.168.201.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4432.532933] LustreError: 324975:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4432.540645] LustreError: 324975:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4432.567220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4432.596768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4434.136550] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4436.643449] LustreError: 326642:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4436.653598] LustreError: 326642:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4437.976511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4438.149497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4440.021744] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4442.278109] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 4442.403339] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4444.306931] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 4445.669853] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4446.215159] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4447.713977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4447.716803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4447.719106] Lustre: Skipped 1 previous similar message [ 4447.721679] Lustre: Skipped 2 previous similar messages [ 4453.550970] LustreError: 328123:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4453.552713] LustreError: 328123:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4453.611776] Lustre: server umount lustre-OST0000 complete [ 4457.954293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4457.955647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4457.960167] Lustre: Skipped 1 previous similar message [ 4457.964864] Lustre: Skipped 3 previous similar messages [ 4469.216113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4469.251447] LustreError: 328325:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4469.253093] LustreError: 328325:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4469.279206] Lustre: server umount lustre-MDT0000 complete [ 4470.421972] LustreError: 323361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699892 with bad export cookie 9904321582928436546 [ 4470.424731] LustreError: 323361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4470.425062] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4470.549513] Lustre: server umount lustre-MDT0001 complete [ 4473.943479] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 4475.048857] Key type lgssc unregistered [ 4475.188404] LNet: 329054:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4475.191235] LNetError: 329054:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4475.201354] LNet: Removed LNI 192.168.201.112@tcp [ 4475.543988] Key type .llcrypt unregistered [ 4475.544957] Key type ._llcrypt unregistered [ 4483.876817] Key type ._llcrypt registered [ 4483.877887] Key type .llcrypt registered [ 4483.908435] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 4484.255143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4484.278332] alg: No test for adler32 (adler32-zlib) [ 4485.117618] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 4485.188652] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 4486.760133] Key type lgssc registered [ 4487.100837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4490.341364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4490.349979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4491.516920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4493.144469] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4496.832810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4497.021604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4498.433472] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4500.565923] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4502.324232] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4506.787467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4506.788970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4506.795253] Lustre: Skipped 1 previous similar message [ 4506.801058] Lustre: Skipped 1 previous similar message [ 4509.834932] LustreError: 331800:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4509.870409] Lustre: server umount lustre-MDT0000 complete [ 4511.610857] LustreError: 331134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699934 with bad export cookie 11589909591429381889 [ 4511.616760] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4511.620406] LustreError: 331134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4511.693620] LustreError: 332004:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4511.695440] LustreError: 332004:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4511.780413] Lustre: server umount lustre-MDT0001 complete [ 4518.678724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4518.838498] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4518.856357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4518.950754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4520.783099] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4524.881025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4524.928998] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4524.947303] Lustre: 332773:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4525.085280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4526.716368] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4529.372546] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4530.885860] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4534.526305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4534.619901] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4534.623796] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4534.627045] LustreError: 334420:0:(tgt_mount.c:2460:server_fill_super()) Unable to start targets: -2 [ 4534.630063] LustreError: 334420:0:(tgt_mount.c:1954:server_put_super()) no obd lustre-OST0000 [ 4534.632848] LustreError: 334420:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4534.691367] Lustre: server umount lustre-OST0000 complete [ 4534.693040] LustreError: 334420:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4538.807329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4538.903707] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4538.966238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4541.334193] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4543.863377] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4546.071738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4547.598736] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4547.642578] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4547.684081] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4547.734192] Lustre: lustre-OST0001: new disk, initializing [ 4547.737465] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4547.787850] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4550.477951] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4553.008155] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4554.246709] LustreError: 336515:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4554.250908] LustreError: 336515:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4554.277908] Lustre: server umount lustre-OST0001 complete [ 4555.758612] LustreError: 334888:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4555.761679] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4555.763393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4555.764789] LustreError: 334888:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4555.768082] Lustre: Skipped 1 previous similar message [ 4560.871856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4560.872919] LustreError: 334889:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4560.875620] Lustre: Skipped 2 previous similar messages [ 4560.886990] LustreError: 334889:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4561.648134] LustreError: 336716:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4561.652023] LustreError: 336716:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4561.743301] Lustre: server umount lustre-OST0000 complete [ 4565.985942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4565.986159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4565.987049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4565.992891] Lustre: Skipped 1 previous similar message [ 4569.659331] LustreError: 336927:0:(obd_class.h:479:obd_check_dev()) Device 24 not setup [ 4569.662262] LustreError: 336927:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4569.688954] Lustre: server umount lustre-MDT0000 complete [ 4570.794081] LustreError: 332765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755699993 with bad export cookie 11589909591429382428 [ 4570.796766] LustreError: 332765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4570.797623] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4570.930876] Lustre: server umount lustre-MDT0001 complete [ 4574.180178] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 4575.288414] Key type lgssc unregistered [ 4575.411303] LNet: 337655:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4575.413230] LNetError: 337655:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4575.424274] LNet: Removed LNI 192.168.201.112@tcp [ 4575.693247] Key type .llcrypt unregistered [ 4575.694200] Key type ._llcrypt unregistered [ 4588.275661] Key type ._llcrypt registered [ 4588.276785] Key type .llcrypt registered [ 4588.320571] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 4594.647983] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 4594.998774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4595.046028] alg: No test for adler32 (adler32-zlib) [ 4595.896402] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 4595.971167] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 4597.552169] Key type lgssc registered [ 4597.979616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4600.810625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4603.535407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4606.223721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4608.501988] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4611.119560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4611.138620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4611.142914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4612.233554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4612.249354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4612.290190] Lustre: lustre-MDT0000: new disk, initializing [ 4612.314850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4612.320364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4613.891834] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4618.505896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4618.533277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4618.570237] Lustre: 341565:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4618.596929] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4618.599896] Lustre: Skipped 1 previous similar message [ 4618.639224] Lustre: lustre-MDT0001: new disk, initializing [ 4618.671580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4618.684410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4618.690684] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4620.152500] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4623.995894] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4625.604722] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4628.780282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4628.807243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4628.927100] Lustre: lustre-OST0000: new disk, initializing [ 4628.930340] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4628.973438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4629.370863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4629.374229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4629.410825] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4631.281904] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4634.843273] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4636.254233] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4636.350822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4638.046406] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4638.180424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4639.716928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4639.719273] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4639.720111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4639.724973] Lustre: Skipped 1 previous similar message [ 4639.729844] LustreError: Skipped 1 previous similar message [ 4644.837470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4644.843307] Lustre: Skipped 2 previous similar messages [ 4645.172198] LustreError: 344476:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4645.268476] Lustre: server umount lustre-OST0000 complete [ 4649.441590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4649.446270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4649.454379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4655.073931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4655.078737] Lustre: Skipped 2 previous similar messages [ 4660.193166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4660.196396] Lustre: Skipped 1 previous similar message [ 4661.216105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4661.251472] LustreError: 344678:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4661.253136] LustreError: 344678:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4661.276177] Lustre: server umount lustre-MDT0000 complete [ 4662.458182] LustreError: 342341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755700084 with bad export cookie 8409047859058233860 [ 4662.461207] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4662.461285] LustreError: 342341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4662.527578] LustreError: 344880:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4662.530089] LustreError: 344880:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4662.604591] Lustre: server umount lustre-MDT0001 complete [ 4665.075685] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 10:28:05 (1755700085) [ 4666.898645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4669.411657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4678.668345] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 4681.981118] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 4685.570247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4688.552406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4691.049716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4693.812915] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4697.114120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4697.158282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4697.306769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4697.325149] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4697.376621] Lustre: lustre-MDT0000: new disk, initializing [ 4697.416272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4697.427268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4699.219135] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4705.053331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4705.086491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4705.126802] Lustre: 348637:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4705.163657] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4705.165022] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4705.167420] Lustre: Skipped 1 previous similar message [ 4705.206562] Lustre: lustre-MDT0001: new disk, initializing [ 4705.241917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4705.257061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4705.263222] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4707.008710] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4710.747623] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4712.411599] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4715.326092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4715.367688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4715.511176] Lustre: lustre-OST0000: new disk, initializing [ 4715.514836] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4715.558795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4718.143102] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4718.215787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4718.220509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4718.268157] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4721.825324] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4723.526870] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4723.621931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4725.268947] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4725.391024] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4728.806049] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4728.807481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4728.813284] Lustre: Skipped 1 previous similar message [ 4728.814639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4728.817984] LustreError: Skipped 1 previous similar message [ 4728.826579] Lustre: Skipped 1 previous similar message [ 4732.398593] LustreError: 351545:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4732.401029] LustreError: 351545:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4732.477138] Lustre: server umount lustre-OST0000 complete [ 4735.969578] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4735.974629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4735.981887] Lustre: Skipped 1 previous similar message [ 4740.299632] LustreError: 351746:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4740.303427] LustreError: 351746:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4740.338537] Lustre: server umount lustre-MDT0000 complete [ 4742.119784] LustreError: 348627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755700164 with bad export cookie 8409047859058234728 [ 4742.125268] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4742.127499] LustreError: 348627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4742.315107] Lustre: server umount lustre-MDT0001 complete [ 4746.109150] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 10:29:26 (1755700166) [ 4749.524346] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 10:29:30 (1755700170) [ 4753.287035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4753.475948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4755.006703] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4758.497762] LustreError: 352961:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4758.501699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4758.507481] LustreError: 352961:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4760.315842] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4762.676765] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4763.769389] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4766.534414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4768.952885] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4771.310179] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4785.138633] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4785.672741] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4786.230933] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4786.777988] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4787.376113] Lustre: Failing over lustre-MDT0000 [ 4787.524044] LustreError: 355411:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4787.525908] LustreError: 355411:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4787.553307] Lustre: server umount lustre-MDT0000 complete [ 4788.194135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4788.195427] LustreError: 352955:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4788.200924] Lustre: Skipped 3 previous similar messages [ 4788.205102] LustreError: 352955:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4792.093490] LustreError: 355100:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4797.207072] LustreError: 352957:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4797.212511] LustreError: 352957:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4800.258524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4800.300334] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4800.385619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4800.388122] Lustre: Skipped 2 previous similar messages [ 4800.412106] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4801.557150] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4802.327642] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4805.605564] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 4805.611408] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4805.627818] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4806.719227] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4807.218448] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4808.998052] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4809.611094] Lustre: Failing over lustre-MDT0000 [ 4809.705712] LustreError: 356484:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4809.707448] LustreError: 356484:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4809.733879] Lustre: server umount lustre-MDT0000 complete [ 4810.723202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4810.723922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4810.724391] LustreError: 352957:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4810.724404] LustreError: 352957:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4810.727733] Lustre: Skipped 2 previous similar messages [ 4820.961038] LustreError: 354109:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4820.964751] LustreError: 354109:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 9 previous similar messages [ 4822.481779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4822.523969] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4822.636112] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4822.807637] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4823.815584] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4827.622189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 4827.623801] Lustre: Skipped 2 previous similar messages [ 4827.628292] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4827.644821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4828.760554] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4829.260785] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4832.738889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4832.739062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4832.739869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4832.739874] Lustre: Skipped 3 previous similar messages [ 4832.748387] Lustre: Skipped 1 previous similar message [ 4837.528733] Lustre: server umount lustre-MDT0000 complete [ 4837.858251] LustreError: 354109:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4837.864698] LustreError: 354109:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4838.896298] LustreError: 352941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755700261 with bad export cookie 8409047859058237381 [ 4838.899158] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4838.901617] LustreError: 352941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4839.022584] Lustre: server umount lustre-MDT0001 complete [ 4850.421276] LustreError: 357908:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4850.425138] LustreError: 357908:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4850.520305] Lustre: server umount lustre-OST0000 complete [ 4857.803223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4859.401592] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4862.609949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4864.601183] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4867.007510] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4868.740942] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4871.833651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4871.991301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4871.994871] Lustre: Skipped 3 previous similar messages [ 4874.515127] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4876.983274] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4883.239236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 4893.667209] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4893.668567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4893.671686] LustreError: Skipped 1 previous similar message [ 4893.681485] Lustre: Skipped 2 previous similar messages [ 4895.694097] Lustre: server umount lustre-OST0000 complete [ 4898.788726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4898.792085] Lustre: Skipped 5 previous similar messages [ 4903.634390] Lustre: server umount lustre-MDT0000 complete [ 4903.905175] LustreError: 358737:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4903.910059] LustreError: 358737:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 4904.849148] LustreError: 358718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755700327 with bad export cookie 8409047859058238116 [ 4904.852317] LustreError: 358718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4904.852938] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4904.986878] Lustre: server umount lustre-MDT0001 complete [ 4908.701319] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 4909.731208] Key type lgssc unregistered [ 4909.852369] LNet: 361918:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4909.857263] LNetError: 361918:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4909.868410] LNet: Removed LNI 192.168.201.112@tcp [ 4910.183615] Key type .llcrypt unregistered [ 4910.186350] Key type ._llcrypt unregistered [ 4922.252543] Key type ._llcrypt registered [ 4922.254216] Key type .llcrypt registered [ 4922.308766] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 4922.641441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4922.692952] alg: No test for adler32 (adler32-zlib) [ 4923.548682] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 4923.643663] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 4925.224201] Key type lgssc registered [ 4925.796661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4929.389476] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4929.403162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4930.588514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4932.265782] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4936.461735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4936.651973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4938.307768] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4940.937516] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4942.595622] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4945.985508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4946.157623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4948.688529] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4950.951561] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4956.396560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 4961.763965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4961.765899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4961.773096] Lustre: Skipped 2 previous similar messages [ 4961.776725] Lustre: Skipped 1 previous similar message [ 4966.350189] LustreError: 365615:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4966.391116] Lustre: server umount lustre-MDT0000 complete [ 4966.883469] LustreError: 363384:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4966.900417] LustreError: 363384:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 4968.149136] LustreError: 363367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755700390 with bad export cookie 3961100517872672306 [ 4968.155732] LustreError: 363367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4968.156195] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4968.244661] LustreError: 365818:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4968.247464] LustreError: 365818:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4968.323183] Lustre: server umount lustre-MDT0001 complete [ 4980.219387] LustreError: 366018:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4980.223138] LustreError: 366018:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4980.332216] Lustre: server umount lustre-OST0000 complete [ 4986.691501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4986.924219] LustreError: 366748:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4986.954742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4988.754648] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4991.970385] LustreError: 366748:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4992.920521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4993.090348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4994.830565] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 4997.003250] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4998.560777] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5001.778095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5001.967176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5004.660238] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5007.209515] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5012.202953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5022.692297] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5022.695555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5022.699921] Lustre: Skipped 2 previous similar messages [ 5022.705267] Lustre: Skipped 2 previous similar messages [ 5026.108139] LustreError: 369077:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5026.110039] LustreError: 369077:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5026.135833] Lustre: server umount lustre-MDT0000 complete [ 5027.305363] LustreError: 366728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755700449 with bad export cookie 3961100517872673412 [ 5027.308711] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5027.311479] LustreError: 366728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5027.441064] Lustre: server umount lustre-MDT0001 complete [ 5038.833853] LustreError: 369482:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5038.836363] LustreError: 369482:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5038.897739] Lustre: server umount lustre-OST0000 complete [ 5043.479866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5043.614962] LustreError: 370309:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5043.634324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5044.842138] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5048.518344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5050.261820] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5052.791677] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5054.499162] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5057.853990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5060.823456] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5063.427838] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5069.350185] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5079.523950] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5079.526808] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5079.531039] LustreError: Skipped 1 previous similar message [ 5079.532713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5079.532722] Lustre: Skipped 1 previous similar message [ 5079.539613] Lustre: Skipped 1 previous similar message [ 5084.647716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5084.651900] Lustre: Skipped 2 previous similar messages [ 5089.761953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5089.763474] Lustre: Skipped 1 previous similar message [ 5090.272143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5090.294326] LustreError: 372559:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5090.298107] LustreError: 372559:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5090.389136] Lustre: server umount lustre-OST0000 complete [ 5094.883620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5094.884305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5098.415277] Lustre: server umount lustre-MDT0000 complete [ 5100.003468] LustreError: 370309:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5100.012702] LustreError: 370309:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5100.148829] LustreError: 370288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755700522 with bad export cookie 3961100517872674406 [ 5100.154854] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5100.156415] LustreError: 370288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5100.331338] Lustre: server umount lustre-MDT0001 complete [ 5105.067766] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 5106.596804] Key type lgssc unregistered [ 5106.752669] LNet: 373489:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5106.759673] LNetError: 373489:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5106.771563] LNet: Removed LNI 192.168.201.112@tcp [ 5107.265248] Key type .llcrypt unregistered [ 5107.267309] Key type ._llcrypt unregistered [ 5120.144268] Key type ._llcrypt registered [ 5120.145783] Key type .llcrypt registered [ 5120.189338] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 5120.465859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5120.472270] alg: No test for adler32 (adler32-zlib) [ 5121.336491] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 5121.435477] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 5123.016165] Key type lgssc registered [ 5123.570510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5127.541152] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5127.553643] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5127.559377] LustreError: 374880:0:(osd_handler.c:8370:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5127.570822] LustreError: 374880:0:(obd_config.c:779:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5127.576086] LustreError: 374880:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5127.581308] LustreError: 374880:0:(tgt_mount.c:2428:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5127.587239] LustreError: 374880:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5131.335095] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5131.340136] LustreError: 375278:0:(osd_handler.c:8370:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5131.353772] LustreError: 375278:0:(obd_config.c:779:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5131.360937] LustreError: 375278:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5131.368859] LustreError: 375278:0:(tgt_mount.c:2428:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5131.376375] LustreError: 375278:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5140.126595] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 5141.587727] Key type lgssc unregistered [ 5141.754644] LNet: 376257:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5141.759630] LNetError: 376257:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5141.772548] LNet: Removed LNI 192.168.201.112@tcp [ 5142.171712] Key type .llcrypt unregistered [ 5142.173884] Key type ._llcrypt unregistered [ 5150.215295] Key type ._llcrypt registered [ 5150.216678] Key type .llcrypt registered [ 5150.264239] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 5157.492523] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 5157.794509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5157.807891] alg: No test for adler32 (adler32-zlib) [ 5158.657685] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 5158.748617] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 5160.336168] Key type lgssc registered [ 5160.871036] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5165.471561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5168.664908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5171.585680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5174.736319] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5178.108071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5178.142352] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5178.152976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5179.253530] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5179.267854] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5179.302477] Lustre: lustre-MDT0000: new disk, initializing [ 5179.328084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5179.336083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5180.960739] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5186.494161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5186.534043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5186.590857] Lustre: 379350:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5186.637210] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5186.640799] Lustre: Skipped 1 previous similar message [ 5186.686376] Lustre: lustre-MDT0001: new disk, initializing [ 5186.728288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5186.750161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5186.757477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5188.310495] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5192.094877] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5193.740417] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5196.994072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5197.030101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5197.176760] Lustre: lustre-OST0000: new disk, initializing [ 5197.178793] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5197.206713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5199.838690] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5202.426372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5202.431218] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5202.440024] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 5203.922459] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5205.699801] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5205.788327] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5207.581549] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5207.715122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5210.082900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5210.083210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5210.086026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5210.091009] Lustre: Skipped 1 previous similar message [ 5214.767262] LustreError: 382265:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 5214.842531] Lustre: server umount lustre-OST0000 complete [ 5217.761477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5217.766644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5217.777279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5217.781285] Lustre: Skipped 1 previous similar message [ 5220.322201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5220.323319] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5222.729335] LustreError: 382466:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 5222.733093] LustreError: 382466:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5222.772334] Lustre: server umount lustre-MDT0000 complete [ 5224.587771] LustreError: 381083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755700647 with bad export cookie 11367318690251198442 [ 5224.592668] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5224.594671] LustreError: 381083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5224.693277] LustreError: 382668:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5224.697577] LustreError: 382668:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5224.806590] Lustre: server umount lustre-MDT0001 complete [ 5228.877718] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5229.711206] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 10:37:30 (1755700650) [ 5232.931742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5233.171031] LustreError: 383386:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5233.183401] LustreError: 383386:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5233.211959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5234.907921] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5238.242181] LustreError: 383386:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5238.777624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5238.962344] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5240.618614] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5243.171433] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5245.008685] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5248.378717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5248.538928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5250.863413] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5253.291913] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5256.458901] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5256.495819] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5256.587933] Lustre: lustre-OST0001: new disk, initializing [ 5256.591606] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5259.116625] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5261.858186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5261.864062] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5261.911901] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5263.060314] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5266.922464] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5266.922682] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5266.926977] LustreError: Skipped 1 previous similar message [ 5266.928285] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5266.934038] Lustre: Skipped 1 previous similar message [ 5266.943690] Lustre: Skipped 2 previous similar messages [ 5271.279243] LustreError: 386561:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5271.283994] LustreError: 386561:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5271.313073] Lustre: server umount lustre-OST0001 complete [ 5272.037846] LustreError: 384973:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5272.048656] LustreError: 384973:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5275.106412] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5275.111183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5275.119140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5275.122908] Lustre: Skipped 1 previous similar message [ 5277.161516] LustreError: 384973:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5277.173077] LustreError: 384973:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5279.472886] LustreError: 386763:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5279.476391] LustreError: 386763:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5279.568796] Lustre: server umount lustre-OST0000 complete [ 5287.394122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5287.398146] Lustre: Skipped 5 previous similar messages [ 5287.571084] LustreError: 386965:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5287.575027] LustreError: 386965:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5287.623654] Lustre: server umount lustre-MDT0000 complete [ 5289.347224] LustreError: 383365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755700711 with bad export cookie 11367318690251199310 [ 5289.354311] LustreError: 383365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5289.355177] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5295.612870] Lustre: server umount lustre-MDT0001 complete [ 5300.256412] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 5301.244232] Key type lgssc unregistered [ 5301.365325] LNet: 387696:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5301.367422] LNetError: 387696:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5301.382262] LNet: Removed LNI 192.168.201.112@tcp [ 5301.661064] Key type .llcrypt unregistered [ 5301.662068] Key type ._llcrypt unregistered [ 5312.066544] Key type ._llcrypt registered [ 5312.067514] Key type .llcrypt registered [ 5312.099207] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 5318.911051] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 5319.329595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5319.336780] alg: No test for adler32 (adler32-zlib) [ 5320.171381] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 5320.243098] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 5321.816138] Key type lgssc registered [ 5322.123609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5325.003682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5327.274066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5329.130289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5331.079807] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5333.204850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5333.225175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5333.230473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5334.308466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5334.318852] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5334.352403] Lustre: lustre-MDT0000: new disk, initializing [ 5334.374971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5334.380374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5335.510645] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5339.466593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5339.490630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5339.523489] Lustre: 391602:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5339.553190] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5339.554797] Lustre: Skipped 1 previous similar message [ 5339.588045] Lustre: lustre-MDT0001: new disk, initializing [ 5339.623531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5339.634319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5339.637434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5340.787993] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5343.739641] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5345.030527] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5347.299365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5347.326515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5347.438137] Lustre: lustre-OST0000: new disk, initializing [ 5347.440165] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5347.477737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5349.391673] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5350.396788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5350.399801] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5350.435178] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5352.336360] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5353.492148] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5353.553181] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5354.735133] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5354.814652] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5355.491937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5355.492587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5355.493957] LustreError: Skipped 1 previous similar message [ 5355.497131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5355.498989] Lustre: Skipped 1 previous similar message [ 5360.612935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5360.614509] Lustre: Skipped 1 previous similar message [ 5361.519321] LustreError: 394510:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5361.591108] Lustre: server umount lustre-OST0000 complete [ 5365.216908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5365.218741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5365.221453] Lustre: Skipped 1 previous similar message [ 5365.222861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5368.954638] LustreError: 394712:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5368.957308] LustreError: 394712:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5368.981892] Lustre: server umount lustre-MDT0000 complete [ 5370.144260] LustreError: 394713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755700792 with bad export cookie 4168900522091240409 [ 5370.148237] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5370.151316] LustreError: 394713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5370.222899] LustreError: 394914:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5370.224499] LustreError: 394914:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5370.323235] Lustre: server umount lustre-MDT0001 complete [ 5373.101964] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 10:39:53 (1755700793) [ 5375.803751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5378.361052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5378.489513] Lustre: lustre-MDT0000: new disk, initializing [ 5378.519569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5379.715510] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5382.354872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5383.635657] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5385.362735] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5386.446421] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5387.132875] LustreError: 396208:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5387.138589] LustreError: 396208:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5387.145233] LustreError: 396208:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 5387.746306] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5387.746435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5387.747200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5387.747207] Lustre: Skipped 1 previous similar message [ 5387.753635] Lustre: Skipped 2 previous similar messages [ 5391.841824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5391.843359] Lustre: Skipped 2 previous similar messages [ 5393.280869] LustreError: 397510:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5393.284597] LustreError: 397510:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5393.323472] Lustre: server umount lustre-MDT0000 complete [ 5394.671153] LustreError: 396158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755700817 with bad export cookie 4168900522091241277 [ 5394.676161] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5394.832257] Lustre: server umount lustre-MDT0001 complete [ 5398.209586] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 10:40:18 (1755700818) [ 5400.556799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5400.709084] LustreError: 398430:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5400.719841] LustreError: 398430:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5400.730521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5400.732182] Lustre: Skipped 1 previous similar message [ 5402.126854] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5405.184508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5406.642164] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5408.355288] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5409.470956] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5411.753474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5411.813879] LustreError: 400016:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5411.884084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5411.888071] Lustre: Skipped 1 previous similar message [ 5413.744252] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5415.445772] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5419.183164] Lustre: Permanently deactivating lustre-OST0000 [ 5419.189521] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5422.064094] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5422.355270] LustreError: 400959:0:(mgs_llog.c:1937:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5422.360626] LustreError: 400959:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 5422.364210] LustreError: 400959:0:(mgs_handler.c:1161:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5422.910496] LustreError: 401008:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 5422.912327] LustreError: 401008:0:(mgs_handler.c:1161:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5429.550089] LustreError: 401105:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5429.552816] LustreError: 401105:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5429.618102] Lustre: server umount lustre-OST0000 complete [ 5430.405048] LustreError: 401257:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 5430.407017] LustreError: 401257:0:(mgs_handler.c:1161:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5431.264785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5431.266824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5431.270226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5435.873824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5440.996739] LustreError: 401354:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5440.998597] LustreError: 401354:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5441.023445] Lustre: server umount lustre-MDT0000 complete [ 5442.148964] LustreError: 400761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755700864 with bad export cookie 4168900522091241809 [ 5442.151526] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5442.152746] LustreError: 400761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5451.815600] Lustre: server umount lustre-MDT0001 complete [ 5454.360510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5455.466662] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5456.267563] LustreError: 402391:0:(mgs_handler.c:1161:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5458.400528] Lustre: 402729:0:(mgs_llog.c:1698:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.112@tcp [ 5459.107565] LustreError: 402828:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5459.109224] LustreError: 402828:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5459.134700] Lustre: server umount MGS complete [ 5463.093975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5463.264718] LustreError: 403455:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5463.269367] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5463.276698] Lustre: Skipped 1 previous similar message [ 5463.301872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5464.613284] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5467.455191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5467.549263] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5469.023202] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5470.703502] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5471.928361] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5472.631626] Lustre: Permanently reactivating lustre-OST0000 [ 5472.635635] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5472.637315] Lustre: Skipped 2 previous similar messages [ 5497.824303] LustreError: 404997:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5500.583221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5500.710985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5500.715279] Lustre: Skipped 1 previous similar message [ 5502.640430] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5504.322773] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5517.280755] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5517.282686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5517.285943] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5517.288887] Lustre: Skipped 4 previous similar messages [ 5522.285315] LustreError: 406775:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5522.286929] LustreError: 406775:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5522.345044] Lustre: server umount lustre-OST0000 complete [ 5529.553570] Lustre: server umount lustre-MDT0000 complete [ 5530.689177] LustreError: 403436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755700953 with bad export cookie 4168900522091242754 [ 5530.692467] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5530.694855] LustreError: 403436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5530.839329] Lustre: server umount lustre-MDT0001 complete [ 5534.067279] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 5535.000425] Key type lgssc unregistered [ 5535.112356] LNet: 407707:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5535.114277] LNetError: 407707:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5535.123272] LNet: Removed LNI 192.168.201.112@tcp [ 5535.390902] Key type .llcrypt unregistered [ 5535.392304] Key type ._llcrypt unregistered [ 5541.268585] Key type ._llcrypt registered [ 5541.269623] Key type .llcrypt registered [ 5541.301709] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 5546.911763] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 5547.190031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5547.195698] alg: No test for adler32 (adler32-zlib) [ 5548.033742] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 5548.112663] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 5549.688126] Key type lgssc registered [ 5550.006970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5552.543129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5554.471728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5556.211192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5558.069336] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5560.157875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5560.178816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5560.183459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5561.260897] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5561.271401] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5561.307192] Lustre: lustre-MDT0000: new disk, initializing [ 5561.329874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5561.335573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5562.453628] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5566.254542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5566.277436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5566.305989] Lustre: 410796:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5566.325805] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5566.327372] Lustre: Skipped 1 previous similar message [ 5566.360974] Lustre: lustre-MDT0001: new disk, initializing [ 5566.393237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5566.409645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5566.414733] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5567.512420] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5570.336290] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5571.373500] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5573.327348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5573.348974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5573.444730] Lustre: lustre-OST0000: new disk, initializing [ 5573.446844] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5573.473378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5575.175928] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5578.156464] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5579.261804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5579.266450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5579.308712] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5579.652407] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5579.736118] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5581.069575] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5581.156816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5584.356252] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5584.357356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5584.358158] LustreError: Skipped 1 previous similar message [ 5584.362085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5584.363970] Lustre: Skipped 1 previous similar message [ 5587.887610] LustreError: 413703:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5587.965677] Lustre: server umount lustre-OST0000 complete [ 5589.473226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5589.473762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5589.476128] Lustre: Skipped 2 previous similar messages [ 5594.593193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5594.594818] Lustre: Skipped 2 previous similar messages [ 5595.321979] LustreError: 413904:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5595.323513] LustreError: 413904:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5595.348162] Lustre: server umount lustre-MDT0000 complete [ 5596.499129] LustreError: 410788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755701019 with bad export cookie 9999648421815331937 [ 5596.501990] LustreError: 410788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5596.502618] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5596.562518] LustreError: 414106:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5596.564257] LustreError: 414106:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5596.644148] Lustre: server umount lustre-MDT0001 complete [ 5599.108834] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5599.702935] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 10:43:40 (1755701020) [ 5601.945283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5602.104505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5603.200690] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5605.811814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5607.178389] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5608.804684] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5609.872521] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5612.065887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5612.167748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5612.169537] Lustre: Skipped 1 previous similar message [ 5613.882682] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5615.770979] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5616.530454] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5627.221231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5631.969814] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5631.970687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5631.971700] LustreError: Skipped 1 previous similar message [ 5631.971780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5631.977127] Lustre: Skipped 1 previous similar message [ 5634.605435] LustreError: 417262:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5634.607319] LustreError: 417262:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5634.671011] Lustre: server umount lustre-OST0000 complete [ 5636.576748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5636.578628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5636.581992] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5636.583943] Lustre: Skipped 1 previous similar message [ 5647.329158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5647.330782] Lustre: Skipped 5 previous similar messages [ 5650.400103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5650.429601] LustreError: 417465:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5650.431247] LustreError: 417465:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5650.457146] Lustre: server umount lustre-MDT0000 complete [ 5651.606131] LustreError: 414804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755701074 with bad export cookie 9999648421815332791 [ 5651.609530] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5651.749784] Lustre: server umount lustre-MDT0001 complete [ 5654.899642] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 5655.834245] Key type lgssc unregistered [ 5655.956300] LNet: 418196:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5655.959145] LNetError: 418196:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5655.968305] LNet: Removed LNI 192.168.201.112@tcp [ 5656.254225] Key type .llcrypt unregistered [ 5656.255169] Key type ._llcrypt unregistered [ 5663.714190] Key type ._llcrypt registered [ 5663.715078] Key type .llcrypt registered [ 5663.747953] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 5663.843178] Key type .llcrypt unregistered [ 5663.844179] Key type ._llcrypt unregistered [ 5670.113910] Key type ._llcrypt registered [ 5670.114841] Key type .llcrypt registered [ 5670.144207] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 5670.515839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5670.527986] alg: No test for adler32 (adler32-zlib) [ 5671.393389] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 5671.490890] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 5673.064110] Key type lgssc registered [ 5673.368704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5675.582787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5675.589259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5676.734054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5677.809404] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5680.297608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5680.404369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5682.021321] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5683.682399] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5685.729405] LustreError: 420331:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5685.734667] LustreError: 420331:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 5685.886605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5685.998449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5687.018800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5687.213628] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5691.361353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5691.364607] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5691.369881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5692.387664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5692.388210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5692.392119] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5692.393480] Lustre: Skipped 1 previous similar message [ 5696.303255] LustreError: 422370:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 5696.369108] Lustre: server umount lustre-OST0000 complete [ 5701.090343] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5701.094226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5701.097634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5703.738187] LustreError: 422573:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5703.739863] LustreError: 422573:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5703.765070] Lustre: server umount lustre-MDT0000 complete [ 5705.009125] LustreError: 420311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755701127 with bad export cookie 10807445711952144701 [ 5705.011615] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5705.012054] LustreError: 420311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5705.073298] LustreError: 422775:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5705.074971] LustreError: 422775:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5705.150121] Lustre: server umount lustre-MDT0001 complete [ 5709.098541] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 5710.037243] Key type lgssc unregistered [ 5710.168410] LNet: 423301:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5710.170506] LNetError: 423301:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5710.180272] LNet: Removed LNI 192.168.201.112@tcp [ 5710.476224] Key type .llcrypt unregistered [ 5710.477183] Key type ._llcrypt unregistered [ 5719.672943] Key type ._llcrypt registered [ 5719.673898] Key type .llcrypt registered [ 5719.705438] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 5720.003693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5720.046090] alg: No test for adler32 (adler32-zlib) [ 5720.882147] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 5720.954349] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 5722.528103] Key type lgssc registered [ 5722.823594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5725.219941] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5725.227791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5742.752249] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5748.896147] LustreError: 424640:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.112@tcp: failed processing log, type 4: rc = -110 [ 5778.592215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5781.408482] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5784.178589] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5787.595481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5787.864376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5787.890141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5789.491586] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5793.520992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5793.699754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5795.356598] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5797.945225] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5799.693282] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5803.489491] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5803.494141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5803.502282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5804.003086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5804.007796] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5804.016650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5807.010344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5807.013207] Lustre: Skipped 1 previous similar message [ 5808.433918] LustreError: 426904:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 5808.498796] Lustre: server umount lustre-OST0000 complete [ 5812.195291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5812.196893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5812.204533] Lustre: Skipped 1 previous similar message [ 5815.820179] LustreError: 427108:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 5815.823963] LustreError: 427108:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5815.858510] Lustre: server umount lustre-MDT0000 complete [ 5817.315138] LustreError: 425562:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5817.329280] LustreError: 425562:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 5817.360075] LustreError: 424649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755701239 with bad export cookie 17003251540907855625 [ 5817.366812] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5817.370346] LustreError: 424649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5817.457500] LustreError: 427311:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5817.461089] LustreError: 427311:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5817.548635] Lustre: server umount lustre-MDT0001 complete [ 5821.451754] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 5822.636140] Key type lgssc unregistered [ 5822.758496] LNet: 427836:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5822.762061] LNetError: 427836:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5822.770484] LNet: Removed LNI 192.168.201.112@tcp [ 5823.142401] Key type .llcrypt unregistered [ 5823.143456] Key type ._llcrypt unregistered [ 5831.513184] Key type ._llcrypt registered [ 5831.514708] Key type .llcrypt registered [ 5831.545883] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 5831.851906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5831.869284] alg: No test for adler32 (adler32-zlib) [ 5832.730699] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 5832.820651] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 5834.392136] Key type lgssc registered [ 5834.701630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5837.140312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5837.146906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5838.322476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5839.878932] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5842.636378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5842.749693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5844.239532] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5846.428613] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5847.953669] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5850.239114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5850.344794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5852.369306] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5854.788738] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5862.638607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5868.004286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5868.007914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5868.013781] Lustre: Skipped 2 previous similar messages [ 5868.021445] Lustre: Skipped 2 previous similar messages [ 5870.368225] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.12@tcp (stopping) [ 5872.135213] LustreError: 431452:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5872.168719] Lustre: server umount lustre-MDT0000 complete [ 5873.100644] LustreError: 429186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.12@tcp arrived at 1755701295 with bad export cookie 12072037847322348677 [ 5873.104986] LustreError: 429186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5873.977933] LustreError: 429186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755701296 with bad export cookie 12072037847322348026 [ 5873.986610] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5874.145719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5874.150335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5874.159664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5874.163121] Lustre: Skipped 3 previous similar messages [ 5876.192814] LustreError: 431226:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5876.193307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5876.200134] LustreError: 431226:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 5878.241854] LustreError: 431226:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5878.251430] LustreError: 431226:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5880.050190] LustreError: 431653:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5880.054128] LustreError: 431653:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5880.154339] Lustre: server umount lustre-OST0000 complete [ 5882.307892] LustreError: 432007:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5882.309751] LustreError: 432007:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5882.383081] Lustre: server umount lustre-MDT0001 complete [ 5885.557493] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 5886.534923] Key type lgssc unregistered [ 5886.662413] LNet: 432533:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5886.664386] LNetError: 432533:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5886.672284] LNet: Removed LNI 192.168.201.112@tcp [ 5886.946043] Key type .llcrypt unregistered [ 5886.947020] Key type ._llcrypt unregistered [ 5895.934786] Key type ._llcrypt registered [ 5895.936513] Key type .llcrypt registered [ 5895.971372] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 5896.244318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5896.250032] alg: No test for adler32 (adler32-zlib) [ 5897.107872] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 5897.183392] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 5898.760104] Key type lgssc registered [ 5899.209444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5902.372358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5902.387231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5903.608405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5905.452996] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5909.579856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5909.730925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5911.523537] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5913.938472] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5915.619166] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5918.878496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5919.072822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5921.657097] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 5924.163810] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5929.326475] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5934.564410] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5934.567776] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5934.577858] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5934.581798] Lustre: Skipped 1 previous similar message [ 5939.683926] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5939.688325] Lustre: Skipped 2 previous similar messages [ 5944.805918] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5944.810483] Lustre: Skipped 2 previous similar messages [ 5945.312129] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5945.363972] LustreError: 436129:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5945.393301] Lustre: server umount lustre-MDT0001 complete [ 5949.923685] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5949.923963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5949.924808] LustreError: 433902:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5949.924831] LustreError: 433902:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5949.949934] Lustre: Skipped 1 previous similar message [ 5949.953677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5949.957120] Lustre: Skipped 1 previous similar message [ 5953.523125] LustreError: 436334:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5953.525584] LustreError: 436334:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5953.616371] Lustre: server umount lustre-OST0000 complete [ 5955.042238] LustreError: 433901:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5955.398124] LustreError: 436535:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5955.401769] LustreError: 436535:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5955.520554] Lustre: server umount lustre-MDT0000 complete [ 5961.647552] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 5963.291735] Key type lgssc unregistered [ 5963.447876] LNet: 437213:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5963.453357] LNetError: 437213:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5963.467718] LNet: Removed LNI 192.168.201.112@tcp [ 5963.962773] Key type .llcrypt unregistered [ 5963.964915] Key type ._llcrypt unregistered [ 5975.287236] Key type ._llcrypt registered [ 5975.289566] Key type .llcrypt registered [ 5975.349282] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 5982.237181] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 5982.654885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5982.664555] alg: No test for adler32 (adler32-zlib) [ 5983.512731] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 5983.587264] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 5985.160126] Key type lgssc registered [ 5985.492144] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5989.690984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5992.628665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5995.011510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5997.988916] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6002.150712] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 6005.960342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6009.059080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6009.096951] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6009.107663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6010.196790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6010.208120] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6010.241903] Lustre: lustre-MDT0000: new disk, initializing [ 6010.265881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6010.273072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6011.903363] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6016.074969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6019.237176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6019.279153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6019.450395] Lustre: lustre-OST0000: new disk, initializing [ 6019.454150] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6019.457849] Lustre: Skipped 1 previous similar message [ 6019.504856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6022.018959] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6026.088634] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6027.796348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6027.804983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6027.822306] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6035.981076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6039.262130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6039.301988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6039.350099] Lustre: 441140:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6039.383562] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6039.438659] Lustre: lustre-MDT0001: new disk, initializing [ 6039.493579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6039.510977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6039.514994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6041.095683] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6045.283540] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6045.394351] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6047.170400] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6048.326989] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [ 6052.837179] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6052.839915] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6052.843918] Lustre: Skipped 1 previous similar message [ 6052.844318] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6052.847357] Lustre: Skipped 1 previous similar message [ 6053.664300] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.12@tcp (stopping) [ 6056.845398] LustreError: 444538:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6056.880091] Lustre: server umount lustre-MDT0001 complete [ 6063.075303] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6063.076387] LustreError: 444451:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6063.083438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6063.092802] LustreError: 444451:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 6065.457578] LustreError: 444789:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 6065.461364] LustreError: 444789:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6065.560735] Lustre: server umount lustre-OST0000 complete [ 6073.235862] LustreError: 444990:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6073.239552] LustreError: 444990:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6073.361253] Lustre: server umount lustre-MDT0000 complete [ 6079.027814] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 6080.612230] Key type lgssc unregistered [ 6080.754598] LNet: 445669:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6080.758260] LNetError: 445669:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6080.766477] LNet: Removed LNI 192.168.201.112@tcp [ 6081.184028] Key type .llcrypt unregistered [ 6081.185062] Key type ._llcrypt unregistered [ 6099.380162] Key type ._llcrypt registered [ 6099.382025] Key type .llcrypt registered [ 6099.437712] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 6099.588613] Key type .llcrypt unregistered [ 6099.590819] Key type ._llcrypt unregistered [ 6107.459710] Key type ._llcrypt registered [ 6107.460671] Key type .llcrypt registered [ 6107.498625] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 6107.803970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6107.839147] alg: No test for adler32 (adler32-zlib) [ 6108.701433] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 6108.802601] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 6110.376200] Key type lgssc registered [ 6110.950229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6114.777786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6117.843176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6120.823364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6123.777136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6123.805639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6123.814026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6124.914039] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6124.928152] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6124.971748] Lustre: lustre-MDT0000: new disk, initializing [ 6125.000600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6125.010918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6126.772873] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6132.232786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6132.274413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6132.334617] Lustre: 448977:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6132.380978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6132.385138] Lustre: Skipped 1 previous similar message [ 6132.440062] Lustre: lustre-MDT0001: new disk, initializing [ 6132.491075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6132.515639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6132.524272] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6134.151563] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6138.240428] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6140.120159] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6143.358957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6143.399733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6143.519753] Lustre: lustre-OST0000: new disk, initializing [ 6143.522716] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6143.563312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6145.980549] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6149.415821] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6149.628595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6149.631605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6149.673769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6158.304805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6158.306789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6158.310260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6159.842124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6159.843224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6159.845934] Lustre: Skipped 1 previous similar message [ 6159.847366] Lustre: Skipped 1 previous similar message [ 6163.131170] LustreError: 451404:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6163.157758] Lustre: server umount lustre-MDT0000 complete [ 6164.221112] LustreError: 449754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755701586 with bad export cookie 2601584441908994412 [ 6164.224312] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6164.225889] LustreError: 449754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6164.286047] LustreError: 451607:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6164.287704] LustreError: 451607:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6164.356126] Lustre: server umount lustre-MDT0001 complete [ 6174.578458] LustreError: 451807:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6174.580057] LustreError: 451807:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6174.642263] Lustre: server umount lustre-OST0000 complete [ 6177.922956] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 10:53:18 (1755701598) [ 6178.434558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6181.432653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6181.656824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6183.288156] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6186.978275] LustreError: 452633:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6186.990252] LustreError: 452633:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6187.134271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6187.310272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6188.931927] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6191.322231] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6192.875710] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6195.966486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6195.993151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6196.053590] Lustre: Found index 0 for lustre-OST0000, updating log [ 6196.109522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6198.633992] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6201.059980] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6207.405769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6217.699236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6217.701028] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6217.703580] LustreError: Skipped 1 previous similar message [ 6217.710078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6217.716178] Lustre: Skipped 2 previous similar messages [ 6218.928231] LustreError: 454894:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6218.932232] LustreError: 454894:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6219.042599] Lustre: server umount lustre-OST0000 complete [ 6220.435124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6223.329459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6223.334116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6223.341042] Lustre: Skipped 1 previous similar message [ 6223.344356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6223.347777] Lustre: Skipped 2 previous similar messages [ 6227.529414] LustreError: 455149:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6227.531454] LustreError: 455149:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6227.568969] Lustre: server umount lustre-MDT0000 complete [ 6229.406160] LustreError: 452612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755701651 with bad export cookie 2601584441908995791 [ 6229.414798] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6229.604342] Lustre: server umount lustre-MDT0001 complete [ 6233.783673] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 10:54:14 (1755701654) [ 6234.296690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6234.702054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6235.187334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6235.692122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6236.161603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6236.652764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6237.139401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6237.646150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6238.146495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6238.647075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6239.148644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6239.635195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6240.100782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6240.604908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6241.095812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6241.592815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6242.102220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6242.566245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6243.059733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6243.542346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6244.037822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6244.535416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6245.040481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6245.496639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6245.971178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6246.463679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6246.977901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6247.507093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6248.022331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6248.529528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6249.042734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6249.513570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6249.997915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6250.523729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6251.032512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6251.541711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6252.072886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6252.446422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6252.823618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6253.153516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6253.590547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6253.966399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6254.387225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6254.840700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6255.312087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6255.805516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6256.317830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6256.822626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6257.332153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6257.826324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6258.304974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6261.936105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6262.201726] LustreError: 458736:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6262.250376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6264.073251] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6267.362535] LustreError: 458736:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6268.318427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6270.469568] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6273.104241] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6274.825409] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6278.169605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6278.232411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6278.379160] Lustre: Found index 0 for lustre-OST0000, updating log [ 6281.309595] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6284.024929] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6288.036480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6288.040323] Lustre: Skipped 3 previous similar messages [ 6288.044571] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6288.051631] Lustre: Skipped 1 previous similar message [ 6291.056218] LustreError: 461016:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6291.059957] LustreError: 461016:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6291.166129] Lustre: server umount lustre-OST0000 complete [ 6293.985765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6297.059334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6297.063144] Lustre: Skipped 8 previous similar messages [ 6307.296220] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6307.353315] LustreError: 461226:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6307.359731] LustreError: 461226:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6307.415238] Lustre: server umount lustre-MDT0000 complete [ 6308.321685] LustreError: 458736:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6308.325687] LustreError: 458736:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6309.242687] LustreError: 460392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755701731 with bad export cookie 2601584441908996876 [ 6309.249960] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6309.255110] LustreError: 460392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6309.430718] Lustre: server umount lustre-MDT0001 complete [ 6310.764734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6318.605755] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 6323.526540] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 6328.801863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6331.723727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6334.586586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6336.615125] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6339.282929] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 10:56:00 (1755701760) [ 6345.468892] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 6346.517959] Key type lgssc unregistered [ 6346.650517] LNet: 464972:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6346.652893] LNetError: 464972:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6465.448366] LNet: Removed LNI 192.168.201.112@tcp [ 6465.979316] Key type .llcrypt unregistered [ 6465.981807] Key type ._llcrypt unregistered [ 6474.058151] Key type ._llcrypt registered [ 6474.060293] Key type .llcrypt registered [ 6474.115140] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 6474.554526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6474.584074] alg: No test for adler32 (adler32-zlib) [ 6475.474458] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 6475.561912] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 6477.152229] Key type lgssc registered [ 6477.743776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6480.010442] LNet: Added LNI 192.168.201.112@tcp100 [8/256/0/180] [ 6480.796166] Lustre: DEBUG MARKER: SKIP: conf-sanity test_73c unable to configure net [ 6489.180576] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 6490.699335] Key type lgssc unregistered [ 6490.928385] LNet: 466828:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6490.936381] LNetError: 466828:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6490.941443] LNet: Removed LNI 192.168.201.112@tcp [ 6490.951589] LNet: Removed LNI 192.168.201.112@tcp100 [ 6491.415782] Key type .llcrypt unregistered [ 6491.418399] Key type ._llcrypt unregistered [ 6501.370511] Key type ._llcrypt registered [ 6501.371427] Key type .llcrypt registered [ 6501.405333] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 6509.546688] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 6509.967179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6510.010072] alg: No test for adler32 (adler32-zlib) [ 6510.893454] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 6511.015531] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 6512.624285] Key type lgssc registered [ 6513.283526] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6517.884856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6521.258188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6524.303694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6527.162425] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6530.431121] Lustre: DEBUG MARKER: == conf-sanity test 74: Test per-device adaptive timeout parameters ========================================================== 10:59:11 (1755701951) [ 6533.784092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6533.819941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6533.830630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6534.950146] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6534.969310] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6535.011072] Lustre: lustre-MDT0000: new disk, initializing [ 6535.053526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6535.065227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6536.837283] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6542.087543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6542.114476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6542.142585] Lustre: 470305:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6542.165862] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6542.167284] Lustre: Skipped 1 previous similar message [ 6542.201229] Lustre: lustre-MDT0001: new disk, initializing [ 6542.236175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6542.249124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6542.252148] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6543.384120] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6546.225441] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6547.319699] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6549.435929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6549.459425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6549.572310] Lustre: lustre-OST0000: new disk, initializing [ 6549.574127] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6549.607305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6551.279905] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6554.114762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6554.118272] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6554.160798] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6554.208963] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6558.538638] loop0: detected capacity change from 0 to 204800000 [ 6558.663317] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6558.666285] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6558.669441] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6558.672806] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6558.691599] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6558.708339] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6558.720866] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6558.741070] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6558.764221] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6558.793887] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6560.697600] loop0: detected capacity change from 0 to 204800000 [ 6560.756555] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6560.815653] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6562.400441] loop0: detected capacity change from 0 to 204800000 [ 6564.354666] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6564.376448] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6564.417621] Lustre: fs15246-OST0000: new disk, initializing [ 6564.419311] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 6564.436808] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6566.162590] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6569.809387] loop1: detected capacity change from 0 to 204800000 [ 6571.596442] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6571.622919] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6571.651344] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 6571.675539] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 6571.709847] Lustre: fs15246-MDT0000: new disk, initializing [ 6571.762314] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6571.770248] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6571.810810] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 6572.833900] Lustre: 474850:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@000000004ac8512a x1840986910134272/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1755702011 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 6572.953211] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6581.920226] LustreError: 474850:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 6581.923186] LustreError: 474850:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 6581.925702] LustreError: 474850:0:(osp_precreate.c:461:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 6583.273092] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6583.276662] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 6585.529880] LustreError: 475965:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 6585.553415] Lustre: server umount fs15246-MDT0000 complete [ 6588.445938] LustreError: 476467:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 6588.447788] LustreError: 476467:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6588.461316] Lustre: server umount fs15246-OST0000 complete [ 6593.510542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6593.512112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6593.513730] Lustre: Skipped 1 previous similar message [ 6593.516716] Lustre: Skipped 1 previous similar message [ 6597.676171] LustreError: 476968:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6597.677893] LustreError: 476968:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6597.744343] Lustre: server umount lustre-OST0000 complete [ 6603.747332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6603.747597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6603.747910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6603.750698] Lustre: Skipped 1 previous similar message [ 6604.987806] LustreError: 477170:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6604.989698] LustreError: 477170:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6605.017071] Lustre: server umount lustre-MDT0000 complete [ 6606.175492] LustreError: 470297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755702028 with bad export cookie 16106899298735368956 [ 6606.178683] LustreError: 470297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6606.178792] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6606.327964] Lustre: server umount lustre-MDT0001 complete [ 6610.141666] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 6611.139642] Key type lgssc unregistered [ 6611.274335] LNet: 477897:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6611.276414] LNetError: 477897:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6611.284388] LNet: Removed LNI 192.168.201.112@tcp [ 6611.586410] Key type .llcrypt unregistered [ 6611.587427] Key type ._llcrypt unregistered [ 6619.547561] Key type ._llcrypt registered [ 6619.548561] Key type .llcrypt registered [ 6619.581944] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 6619.975876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6620.039046] alg: No test for adler32 (adler32-zlib) [ 6620.891792] Lustre: Lustre: Build Version: 2.16.57_97_gc8148ab [ 6620.972573] LNet: Added LNI 192.168.201.112@tcp [8/256/0/180] [ 6622.552105] Key type lgssc registered [ 6622.923592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6625.208646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6626.657675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6628.248177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6629.757428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6634.510896] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 6637.390767] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 6640.550053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6642.518817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6644.336677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6646.296179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6648.367695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6648.388453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6648.393321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6649.480283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6649.491980] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6649.528889] Lustre: lustre-MDT0000: new disk, initializing [ 6649.552806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6649.558618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6650.715563] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6654.545389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6654.569022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6654.598548] Lustre: 482290:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6654.624385] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6654.625927] Lustre: Skipped 1 previous similar message [ 6654.663064] Lustre: lustre-MDT0001: new disk, initializing [ 6654.698464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6654.710949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6654.714590] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6655.816134] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6658.650651] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6659.733062] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6661.696726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6661.718911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6661.811288] Lustre: lustre-OST0000: new disk, initializing [ 6661.813026] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6661.838318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6663.459699] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_default_debug -1 all [ 6666.346652] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6667.516412] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6668.792016] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6668.794823] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6668.821791] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6669.618117] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 6670.758246] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6670.819193] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6673.890834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6673.891870] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6673.893142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6673.894189] Lustre: Skipped 1 previous similar message [ 6673.899174] LustreError: Skipped 1 previous similar message [ 6677.485134] LustreError: 485211:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6677.550126] Lustre: server umount lustre-OST0000 complete [ 6679.010459] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6679.011434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6679.013597] Lustre: Skipped 1 previous similar message [ 6679.016415] Lustre: Skipped 2 previous similar messages [ 6684.129842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6684.131582] Lustre: Skipped 1 previous similar message [ 6684.798262] LustreError: 485413:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6684.799876] LustreError: 485413:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6684.830544] Lustre: server umount lustre-MDT0000 complete [ 6685.973630] LustreError: 482281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1755702108 with bad export cookie 329495766219896991 [ 6685.976901] LustreError: 482281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6685.977290] LustreError: MGC192.168.201.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6686.045116] LustreError: 485614:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6686.047970] LustreError: 485614:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6686.146715] Lustre: server umount lustre-MDT0001 complete [ 6694.489975] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing set_hostid [ 6699.055082] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing load_modules_local [ 6704.328253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6708.099282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6711.120672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6714.415954] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6715.206307] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6590 sec ========== 11:02:15 (1755702135) [ 6716.062569] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:02:16 (1755702136) === [ 6717.260930] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:02:18 (1755702138) === [ 6733.953851] Lustre: DEBUG MARKER: oleg112-server.virtnet: executing unload_modules_local [ 6735.244470] Key type lgssc unregistered [ 6735.379687] LNet: 490446:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6735.384213] LNetError: 490446:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6735.395590] LNet: Removed LNI 192.168.201.112@tcp [ 6735.831784] Key type .llcrypt unregistered [ 6735.833274] Key type ._llcrypt unregistered