[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 512604978 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002398] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004021] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007011] pid_max: default: 32768 minimum: 301 [ 0.008155] LSM: Security Framework initializing [ 0.009044] Yama: becoming mindful. [ 0.010031] SELinux: Initializing. [ 0.011078] *** VALIDATE selinux *** [ 0.020830] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025209] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026165] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027115] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029133] *** VALIDATE tmpfs *** [ 0.031032] *** VALIDATE proc *** [ 0.032179] *** VALIDATE cgroup *** [ 0.033008] *** VALIDATE cgroup2 *** [ 0.035338] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038032] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039040] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.043810] debug: unmapping init [mem 0xffffffffad859000-0xffffffffad860fff] [ 0.046237] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048032] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049023] ... version: 2 [ 0.050014] ... bit width: 48 [ 0.051011] ... generic registers: 4 [ 0.052014] ... value mask: 0000ffffffffffff [ 0.053014] ... max period: 00007fffffffffff [ 0.054009] ... fixed-purpose events: 3 [ 0.055009] ... event mask: 000000070000000f [ 0.056364] rcu: Hierarchical SRCU implementation. [ 0.058600] smp: Bringing up secondary CPUs ... [ 0.059510] x86: Booting SMP configuration: [ 0.060015] .... node #0, CPUs: #1 #2 #3 [ 0.067509] smp: Brought up 1 node, 4 CPUs [ 0.069009] smpboot: Max logical packages: 1 [ 0.070009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.161042] node 0 deferred pages initialised in 88ms [ 0.164641] devtmpfs: initialized [ 0.165000] x86/mm: Memory block size: 128MB [ 0.166941] gcov: version magic: 0x41383552 [ 0.171298] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.175086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.178454] pinctrl core: initialized pinctrl subsystem [ 0.180140] [ 0.180937] ************************************************************* [ 0.184013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.186008] ** ** [ 0.189009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.192011] ** ** [ 0.194009] ** This means that this kernel is built to expose internal ** [ 0.197017] ** IOMMU data structures, which may compromise security on ** [ 0.199006] ** your system. ** [ 0.201008] ** ** [ 0.203008] ** If you see this message and you are not debugging the ** [ 0.205008] ** kernel, report this immediately to your vendor! ** [ 0.207010] ** ** [ 0.208009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.209008] ************************************************************* [ 0.211772] NET: Registered protocol family 16 [ 0.213405] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.215057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.218087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.221104] cpuidle: using governor menu [ 0.222000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.224675] PCI: Using configuration type 1 for base access [ 0.227819] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.239100] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.241185] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.245528] cryptd: max_cpu_qlen set to 1000 [ 0.248346] ACPI: Added _OSI(Module Device) [ 0.249008] ACPI: Added _OSI(Processor Device) [ 0.250000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.250000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.250000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.260602] ACPI: Interpreter enabled [ 0.261000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.261000] ACPI: Using IOAPIC for interrupt routing [ 0.261000] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.265455] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.276618] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.279032] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.282011] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.285074] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.290535] acpiphp: Slot [2] registered [ 0.292088] acpiphp: Slot [5] registered [ 0.293096] acpiphp: Slot [6] registered [ 0.295092] acpiphp: Slot [7] registered [ 0.297105] acpiphp: Slot [8] registered [ 0.298083] acpiphp: Slot [9] registered [ 0.299121] acpiphp: Slot [10] registered [ 0.301092] acpiphp: Slot [3] registered [ 0.302063] acpiphp: Slot [4] registered [ 0.304064] acpiphp: Slot [11] registered [ 0.305089] acpiphp: Slot [12] registered [ 0.306125] acpiphp: Slot [13] registered [ 0.308067] acpiphp: Slot [14] registered [ 0.309079] acpiphp: Slot [15] registered [ 0.311074] acpiphp: Slot [16] registered [ 0.312073] acpiphp: Slot [17] registered [ 0.314070] acpiphp: Slot [18] registered [ 0.315080] acpiphp: Slot [19] registered [ 0.317068] acpiphp: Slot [20] registered [ 0.318121] acpiphp: Slot [21] registered [ 0.320079] acpiphp: Slot [22] registered [ 0.321067] acpiphp: Slot [23] registered [ 0.322065] acpiphp: Slot [24] registered [ 0.324078] acpiphp: Slot [25] registered [ 0.325063] acpiphp: Slot [26] registered [ 0.326138] acpiphp: Slot [27] registered [ 0.328070] acpiphp: Slot [28] registered [ 0.329069] acpiphp: Slot [29] registered [ 0.331064] acpiphp: Slot [30] registered [ 0.332100] acpiphp: Slot [31] registered [ 0.334052] PCI host bridge to bus 0000:00 [ 0.335012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.338014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.340013] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.342016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.345018] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.348018] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.350162] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.353019] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.356477] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.367015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.370972] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.372012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.373007] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.375011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.377435] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.380804] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.386055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.388974] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.392907] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.403018] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.408017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.416378] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.425022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.432019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.443017] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.449000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.454015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.459029] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.467018] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.478000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.485016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.492017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.512017] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.521597] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.526020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.530020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.543017] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.552000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.555018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.558014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.578018] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.592283] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.600018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.610024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.621021] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.632000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.632106] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.633000] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.635322] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.637216] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.641280] iommu: Default domain type: Passthrough [ 0.643561] SCSI subsystem initialized [ 0.645097] ACPI: bus type USB registered [ 0.646132] usbcore: registered new interface driver usbfs [ 0.647073] usbcore: registered new interface driver hub [ 0.649072] usbcore: registered new device driver usb [ 0.650209] pps_core: LinuxPPS API ver. 1 registered [ 0.652008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.654047] PTP clock support registered [ 0.655164] EDAC MC: Ver: 3.0.0 [ 0.656066] PCI: Using ACPI for IRQ routing [ 0.663478] NetLabel: Initializing [ 0.666014] NetLabel: domain hash size = 128 [ 0.668008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.671103] NetLabel: unlabeled traffic allowed by default [ 0.673128] vgaarb: loaded [ 0.675607] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.677019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.683004] clocksource: Switched to clocksource kvm-clock [ 0.785728] VFS: Disk quotas dquot_6.6.0 [ 0.787384] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.790316] *** VALIDATE ramfs *** [ 0.791748] *** VALIDATE hugetlbfs *** [ 0.793408] pnp: PnP ACPI init [ 0.795694] pnp: PnP ACPI: found 6 devices [ 0.811771] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.815686] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.817605] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.819296] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.822087] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.824352] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.826782] NET: Registered protocol family 2 [ 0.829078] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.834203] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.838142] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.844309] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.847690] TCP: Hash tables configured (established 65536 bind 65536) [ 0.850861] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.854555] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.857085] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.859762] NET: Registered protocol family 1 [ 0.863176] RPC: Registered named UNIX socket transport module. [ 0.865852] RPC: Registered udp transport module. [ 0.866932] RPC: Registered tcp transport module. [ 0.868449] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.870673] NET: Registered protocol family 44 [ 0.872176] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.874124] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.876216] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.878314] PCI: CLS 0 bytes, default 64 [ 0.879421] Unpacking initramfs... [ 2.514907] debug: unmapping init [mem 0xffff8c333cc54000-0xffff8c333ffbffff] [ 2.520888] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.522718] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.527482] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.137823] Initialise system trusted keyrings [ 3.140023] Key type blacklist registered [ 3.142663] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.151142] zbud: loaded [ 3.153773] *** VALIDATE nfs *** [ 3.154807] *** VALIDATE nfs4 *** [ 3.156172] pstore: using deflate compression [ 3.159310] Platform Keyring initialized [ 3.299307] NET: Registered protocol family 38 [ 3.302322] Key type asymmetric registered [ 3.303590] Asymmetric key parser 'x509' registered [ 3.305558] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.329530] io scheduler mq-deadline registered [ 3.331355] io scheduler kyber registered [ 3.336911] io scheduler bfq registered [ 3.339548] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.343492] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.350152] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.353240] ACPI: Power Button [PWRF] [ 3.448741] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.537866] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.724898] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.852863] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.114848] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.142582] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.176576] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.183115] Non-volatile memory driver v1.3 [ 4.184777] Linux agpgart interface v0.103 [ 4.213068] virtio_blk virtio1: [vda] 133040 512-byte logical blocks (68.1 MB/65.0 MiB) [ 4.216378] vda: detected capacity change from 0 to 68116480 [ 4.231541] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.234841] vdb: detected capacity change from 0 to 1073741824 [ 4.249656] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.253021] vdc: detected capacity change from 0 to 2621440000 [ 4.268331] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.271683] vdd: detected capacity change from 0 to 2621440000 [ 4.287401] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.290656] vde: detected capacity change from 0 to 4294967296 [ 4.305713] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.308902] vdf: detected capacity change from 0 to 4294967296 [ 4.315343] libphy: Fixed MDIO Bus: probed [ 4.349298] usbcore: registered new interface driver usbserial_generic [ 4.352045] usbserial: USB Serial support registered for generic [ 4.354671] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.358981] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.361020] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.363600] mousedev: PS/2 mouse device common for all mice [ 4.367240] rtc_cmos 00:05: RTC can wake from S4 [ 4.371534] rtc_cmos 00:05: registered as rtc0 [ 4.373482] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.374337] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.381571] intel_pstate: CPU model not supported [ 4.386582] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.387045] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.408991] hid: raw HID events driver (C) Jiri Kosina [ 4.412650] usbcore: registered new interface driver usbhid [ 4.414686] usbhid: USB HID core driver [ 4.416650] drop_monitor: Initializing network drop monitor service [ 4.422334] Initializing XFRM netlink socket [ 4.425147] NET: Registered protocol family 10 [ 4.430906] Segment Routing with IPv6 [ 4.432388] NET: Registered protocol family 17 [ 4.434360] mpls_gso: MPLS GSO support [ 4.452250] hrtimer: interrupt took 12077602 ns [ 4.517115] RAS: Correctable Errors collector initialized. [ 4.518863] AVX version of gcm_enc/dec engaged. [ 4.520359] AES CTR mode by8 optimization enabled [ 4.678770] sched_clock: Marking stable (4678633456, 0)->(5736719995, -1058086539) [ 4.683171] registered taskstats version 1 [ 4.685287] Loading compiled-in X.509 certificates [ 4.689598] zswap: loaded using pool lzo/zbud [ 4.718162] Key type big_key registered [ 4.743427] Key type encrypted registered [ 4.747945] ima: No TPM chip found, activating TPM-bypass! [ 4.749985] ima: Allocated hash algorithm: sha1 [ 4.756826] ima: No architecture policies found [ 4.758750] evm: Initialising EVM extended attributes: [ 4.760862] evm: security.selinux [ 4.763225] evm: security.ima [ 4.764637] evm: security.capability [ 4.766734] evm: HMAC attrs: 0x1 [ 4.770708] rtc_cmos 00:05: setting system clock to 2025-08-31 21:30:14 UTC (1756675814) [ 4.779713] debug: unmapping init [mem 0xffffffffae803000-0xffffffffae9fffff] [ 4.783719] debug: unmapping init [mem 0xffffffffad582000-0xffffffffad858fff] [ 4.793252] Write protecting the kernel read-only data: 28672k [ 4.797516] debug: unmapping init [mem 0xffffffffabc03000-0xffffffffabdfffff] [ 4.801480] debug: unmapping init [mem 0xffffffffac514000-0xffffffffac5fffff] [ 4.839094] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.845763] systemd[1]: Detected virtualization kvm. [ 4.847387] systemd[1]: Detected architecture x86-64. [ 4.848988] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.877448] systemd[1]: No hostname configured. [ 4.878542] systemd[1]: Set hostname to . [ 4.879798] random: systemd: uninitialized urandom read (16 bytes read) [ 4.881661] systemd[1]: Initializing machine ID from random generator. [ 5.087889] random: systemd: uninitialized urandom read (16 bytes read) [ 5.091378] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 5.098663] random: systemd: uninitialized urandom read (16 bytes read) [ 5.101500] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 5.112965] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.069896] device-mapper: uevent: version 1.0.3 [ 6.071809] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 7.240537] virtio_net virtio0 ens2: renamed from eth0 [ 7.308369] random: fast init done [ 7.409546] scsi host0: ata_piix [ 7.459794] scsi host1: ata_piix [ 7.462232] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.471614] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.274863] random: crng init done [ 12.276356] random: 7 urandom warning(s) missed due to ratelimiting [ 12.698288] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 14.116942] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.642742] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.119910] SELinux: Disabled at runtime. [ 16.214133] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.224052] systemd[1]: Detected virtualization kvm. [ 16.225849] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.243882] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.250441] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.262327] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.266229] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.269254] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.305547] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.318246] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 17.409737] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 18.218437] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 18.886199] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.920864] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.414476] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.443099] EDAC sbridge: Ver: 1.1.2 [ 22.231792] Key type dns_resolver registered [ 22.784294] NFS: Registering the id_resolver key type [ 22.786119] Key type id_resolver registered [ 22.788523] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg118-server login: [ 47.754376] libcfs: loading out-of-tree module taints kernel. [ 47.765976] Key type ._llcrypt registered [ 47.767013] Key type .llcrypt registered [ 47.806650] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 55.068980] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 55.547934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 55.553071] alg: No test for adler32 (adler32-zlib) [ 56.500356] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 56.723584] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 58.319083] Key type lgssc registered [ 58.856199] Lustre: Echo OBD driver; http://www.lustre.org/ [ 65.090389] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 66.221871] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 69.814971] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 72.285864] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 74.700541] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 79.585663] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 83.610156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 83.629902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 83.638698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 84.720686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 84.735461] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 84.771692] Lustre: lustre-MDT0000: new disk, initializing [ 84.796405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 84.802838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 86.068987] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 91.467780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 91.498869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 91.527972] Lustre: 6500:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 91.541725] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 91.544428] Lustre: Skipped 1 previous similar message [ 91.589665] Lustre: lustre-MDT0001: new disk, initializing [ 91.613727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 91.624544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 91.628305] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 92.966617] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 95.267157] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 98.669203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 98.692108] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 98.793900] Lustre: lustre-OST0000: new disk, initializing [ 98.795866] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 98.797609] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 98.816886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 100.742786] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 103.907425] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 103.910321] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 103.925052] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 106.172453] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 106.195349] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 106.234215] Lustre: lustre-OST0001: new disk, initializing [ 106.235943] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 106.251845] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 108.217219] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 113.134302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 113.138472] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 113.158833] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 114.209917] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 117.702636] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 124.689881] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing check_logdir /tmp/testlogs/ [ 126.192812] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing yml_node [ 127.647466] Lustre: DEBUG MARKER: Client: 2.16.58.1 [ 128.509103] Lustre: DEBUG MARKER: MDS: 2.16.58.1 [ 129.411525] Lustre: DEBUG MARKER: OSS: 2.16.58.1 [ 130.010713] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Aug 31 17:32:18 EDT 2025 [ 135.785473] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 136.297139] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 147.935517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 147.937700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 147.941804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 148.960145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 148.960508] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 148.962122] Lustre: Skipped 1 previous similar message [ 148.966103] Lustre: Skipped 1 previous similar message [ 154.080386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 154.083352] Lustre: Skipped 5 previous similar messages [ 159.199893] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 159.201655] Lustre: Skipped 3 previous similar messages [ 162.271126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 162.300196] LustreError: 12207:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 162.326853] Lustre: server umount lustre-MDT0000 complete [ 163.569300] LustreError: 8411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756675973 with bad export cookie 5210653010870842212 [ 163.570837] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 163.572557] LustreError: 8411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 164.320047] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 164.324861] Lustre: Skipped 2 previous similar messages [ 168.351663] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 168.353695] Lustre: Skipped 3 previous similar messages [ 169.657977] LustreError: 12411:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 169.659407] LustreError: 12411:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 169.733132] Lustre: server umount lustre-MDT0001 complete [ 185.311150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 185.328731] LustreError: 12613:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 185.331053] LustreError: 12613:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 185.349345] Lustre: server umount lustre-OST0000 complete [ 192.553741] LustreError: 12815:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 192.555415] LustreError: 12815:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 192.615926] Lustre: server umount lustre-OST0001 complete [ 194.685541] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 197.891642] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 201.316676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 203.576659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 205.554046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 207.540066] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 209.833805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 209.858462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.939556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 209.951435] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 209.991316] Lustre: lustre-MDT0000: new disk, initializing [ 210.018073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 210.024600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 211.306333] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 215.298540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 215.324284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 215.356829] Lustre: 14994:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 215.462388] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 216.744859] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 219.851265] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 220.977279] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 223.168399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 223.190048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 223.293583] Lustre: lustre-OST0000: new disk, initializing [ 223.294893] Lustre: Skipped 1 previous similar message [ 223.296653] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 223.298497] Lustre: Skipped 2 previous similar messages [ 223.333522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 223.335821] Lustre: Skipped 1 previous similar message [ 224.503512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 224.505908] Lustre: Skipped 1 previous similar message [ 224.507497] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 224.545484] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 225.175975] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 228.272201] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 229.676317] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 229.754973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 231.265651] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 231.363346] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 234.093238] LustreError: 17933:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 234.096763] LustreError: 17933:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 234.167587] Lustre: server umount lustre-OST0000 complete [ 235.999869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 236.002154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 236.006039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 240.097035] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 246.242319] LustreError: 18134:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 246.243927] LustreError: 18134:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 246.265561] Lustre: server umount lustre-MDT0000 complete [ 247.421226] LustreError: 14984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676057 with bad export cookie 5210653010870843731 [ 247.423386] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 247.424758] LustreError: 14984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 258.094630] Lustre: server umount lustre-MDT0001 complete [ 259.359603] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 17:34:28 (1756676068) [ 260.114141] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 260.633926] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 17:34:29 (1756676069) [ 262.688366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 262.842947] LustreError: 19005:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 262.848792] LustreError: 19005:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 5 previous similar messages [ 262.867752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 264.108161] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 266.901882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 268.392879] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 270.112860] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 271.290457] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 273.473292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 275.462355] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 277.151282] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 297.439879] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 297.442442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.448759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 297.451085] Lustre: Skipped 4 previous similar messages [ 299.492693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 301.421115] LustreError: 21273:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 301.422849] LustreError: 21273:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 301.482087] Lustre: server umount lustre-OST0000 complete [ 308.886690] Lustre: server umount lustre-MDT0000 complete [ 309.728148] LustreError: 19023:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 309.734447] LustreError: 19023:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 310.131427] LustreError: 18984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676119 with bad export cookie 5210653010870844613 [ 310.134501] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 310.134868] LustreError: 18984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 310.288994] Lustre: server umount lustre-MDT0001 complete [ 314.010505] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 315.145049] Key type lgssc unregistered [ 315.281332] LNet: 22208:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 315.283569] LNetError: 22208:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 315.292272] LNet: Removed LNI 192.168.201.118@tcp [ 315.636110] Key type .llcrypt unregistered [ 315.637886] Key type ._llcrypt unregistered [ 324.614763] Key type ._llcrypt registered [ 324.616208] Key type .llcrypt registered [ 324.653668] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 324.995215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 325.049509] alg: No test for adler32 (adler32-zlib) [ 325.897771] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 325.978679] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 327.559206] Key type lgssc registered [ 327.907385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 330.560697] loop: module loaded [ 330.582141] loop0: detected capacity change from 0 to 204800000 [ 330.670559] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 330.675477] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 330.796830] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 330.800469] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 330.825784] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 330.851313] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 330.868075] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 330.894461] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 332.937095] loop0: detected capacity change from 0 to 204800000 [ 332.998466] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 333.004512] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 333.010970] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 333.061458] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 334.761129] loop0: detected capacity change from 0 to 204800000 [ 336.529512] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 336.546271] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 336.550362] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 337.629285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 337.640243] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 337.676603] Lustre: lustre-MDT0000: new disk, initializing [ 337.701610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 337.708145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 338.941399] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 342.566544] loop1: detected capacity change from 0 to 204800000 [ 344.449448] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 344.473531] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 344.591732] Lustre: lustre-OST07c6: new disk, initializing [ 344.594940] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 344.598094] Lustre: Skipped 1 previous similar message [ 344.623588] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 345.282680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 345.285349] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 345.296414] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 346.570107] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 430.573261] LustreError: 26552:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 430.633717] Lustre: server umount lustre-OST07c6 complete [ 443.362664] LustreError: 27051:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 443.366046] LustreError: 27051:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 443.448847] Lustre: server umount lustre-MDT0000 complete [ 449.080418] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 450.169477] Key type lgssc unregistered [ 450.301371] LNet: 27885:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 450.303796] LNetError: 27885:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 450.312310] LNet: Removed LNI 192.168.201.118@tcp [ 450.630269] Key type .llcrypt unregistered [ 450.632187] Key type ._llcrypt unregistered [ 458.344053] Key type ._llcrypt registered [ 458.345446] Key type .llcrypt registered [ 458.383515] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 458.755137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 458.817964] alg: No test for adler32 (adler32-zlib) [ 459.695238] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 459.789119] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 461.375174] Key type lgssc registered [ 461.736545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 464.443782] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 464.450304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.609151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 466.964750] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 469.978897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 470.109164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 471.563432] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 473.493507] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 474.809276] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 477.042155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 477.160247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 478.849252] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 480.509784] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 491.493920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 497.885672] LustreError: 31332:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 500.704536] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 500.706319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 500.709617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 501.728855] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 501.729885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 501.730237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 501.739517] Lustre: Skipped 1 previous similar message [ 506.539890] LustreError: 31583:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 506.600581] Lustre: server umount lustre-OST0000 complete [ 510.944046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 510.946653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 510.950105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 514.108769] LustreError: 31785:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 514.112191] LustreError: 31785:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 514.139234] Lustre: server umount lustre-MDT0000 complete [ 515.383986] LustreError: 30733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676325 with bad export cookie 3033152452490710865 [ 515.389150] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 515.455347] LustreError: 31987:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 515.457011] LustreError: 31987:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 515.562921] Lustre: server umount lustre-MDT0001 complete [ 519.115692] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 520.176868] Key type lgssc unregistered [ 520.311432] LNet: 32517:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 520.315459] LNetError: 32517:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 520.326284] LNet: Removed LNI 192.168.201.118@tcp [ 520.617418] Key type .llcrypt unregistered [ 520.618462] Key type ._llcrypt unregistered [ 528.015557] Key type ._llcrypt registered [ 528.016613] Key type .llcrypt registered [ 528.053389] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 528.384935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 528.421870] alg: No test for adler32 (adler32-zlib) [ 529.282852] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 529.366052] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 530.951159] Key type lgssc registered [ 531.359100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 533.905925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 533.914677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 535.083884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 536.462936] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 539.282287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 539.415787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 540.728660] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 542.497821] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 543.700402] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 545.969626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 546.083494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 548.022888] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 549.849622] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 559.399560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 564.704493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 564.706555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 564.707094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 564.709541] Lustre: Skipped 1 previous similar message [ 566.253428] LustreError: 36015:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 566.311700] Lustre: server umount lustre-OST0000 complete [ 569.596749] LustreError: 36215:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 569.599377] LustreError: 36215:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 569.625553] Lustre: server umount lustre-MDT0000 complete [ 569.824538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 569.825218] LustreError: 33740:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 569.827654] Lustre: Skipped 1 previous similar message [ 569.837801] LustreError: 33740:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 570.849631] LustreError: 33726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676380 with bad export cookie 4073096862853033774 [ 570.852911] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 570.854886] LustreError: 33726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 570.914714] LustreError: 36416:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 570.917021] LustreError: 36416:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 571.001069] Lustre: server umount lustre-MDT0001 complete [ 574.760091] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 575.884511] Key type lgssc unregistered [ 576.019402] LNet: 36947:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 576.021715] LNetError: 36947:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 576.034330] LNet: Removed LNI 192.168.201.118@tcp [ 576.318704] Key type .llcrypt unregistered [ 576.319754] Key type ._llcrypt unregistered [ 585.004166] Key type ._llcrypt registered [ 585.005151] Key type .llcrypt registered [ 585.038729] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 585.350051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 585.388062] alg: No test for adler32 (adler32-zlib) [ 586.246355] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 586.335127] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 587.911129] Key type lgssc registered [ 588.271731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 590.757623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 590.765028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 591.926145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 593.239597] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 596.256625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 596.379077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 597.712934] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 599.564823] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 600.777872] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 603.142987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 603.260903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 605.223472] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 607.076734] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 616.620264] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 622.052497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 622.053222] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 622.054754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 622.060970] Lustre: Skipped 2 previous similar messages [ 625.721984] LustreError: 40586:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 625.751123] Lustre: server umount lustre-MDT0000 complete [ 626.989806] LustreError: 38301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676436 with bad export cookie 62384688504906427 [ 626.995367] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 627.168111] LustreError: 38316:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 627.168380] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 627.176151] LustreError: 38316:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 627.180388] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 627.189335] Lustre: Skipped 3 previous similar messages [ 631.199826] LustreError: 38314:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 631.200208] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 631.203906] LustreError: 38314:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 631.205482] Lustre: Skipped 1 previous similar message [ 632.711459] LustreError: 38314:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 633.147566] LustreError: 40789:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 633.150747] LustreError: 40789:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 633.222806] Lustre: server umount lustre-MDT0001 complete [ 651.117696] LustreError: 41012:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 651.119867] LustreError: 41012:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 651.186074] Lustre: server umount lustre-OST0000 complete [ 656.839194] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 657.931495] Key type lgssc unregistered [ 658.065398] LNet: 41848:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 658.067776] LNetError: 41848:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 658.075290] LNet: Removed LNI 192.168.201.118@tcp [ 658.397879] Key type .llcrypt unregistered [ 658.399897] Key type ._llcrypt unregistered [ 667.311510] Key type ._llcrypt registered [ 667.312614] Key type .llcrypt registered [ 667.358139] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 667.754622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 667.827076] alg: No test for adler32 (adler32-zlib) [ 668.682136] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 668.768663] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 670.351203] Key type lgssc registered [ 670.769944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 673.536757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 673.545226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 674.743251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 676.081742] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 678.986543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 679.118498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 680.432278] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 682.196523] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 683.363104] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 685.580532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 685.690333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 687.579751] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 689.479148] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 701.036066] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 709.599860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 709.602422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 709.606185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 711.649773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 711.649804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 711.652981] Lustre: Skipped 1 previous similar message [ 711.656292] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 715.144234] Lustre: lustre-OST0000: Not available for connect from 192.168.201.18@tcp (stopping) [ 720.264323] Lustre: lustre-OST0000: Not available for connect from 192.168.201.18@tcp (stopping) [ 720.266918] Lustre: Skipped 2 previous similar messages [ 721.375106] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 721.394115] LustreError: 45493:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 721.475862] Lustre: server umount lustre-OST0000 complete [ 734.230725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 734.231697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 734.234034] Lustre: Skipped 1 previous similar message [ 740.411386] LustreError: 45871:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 740.413081] LustreError: 45871:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 740.444289] Lustre: server umount lustre-MDT0000 complete [ 741.687642] LustreError: 44843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676551 with bad export cookie 3146870928407463703 [ 741.690883] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 741.691181] LustreError: 44843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 747.841738] LustreError: 46074:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 747.843721] LustreError: 46074:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 747.939778] Lustre: server umount lustre-MDT0001 complete [ 751.529334] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 752.602310] Key type lgssc unregistered [ 752.741348] LNet: 46606:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 752.743499] LNetError: 46606:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 752.752440] LNet: Removed LNI 192.168.201.118@tcp [ 753.086254] Key type .llcrypt unregistered [ 753.088084] Key type ._llcrypt unregistered [ 761.722907] Key type ._llcrypt registered [ 761.724101] Key type .llcrypt registered [ 761.762610] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 762.069533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 762.081054] alg: No test for adler32 (adler32-zlib) [ 762.935110] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 763.012239] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 764.591106] Key type lgssc registered [ 764.962114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 767.594955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 767.603454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 768.769155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 770.069594] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 773.011661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 773.145990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 774.499699] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 776.196854] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 777.416025] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 779.752777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 779.873790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 781.835780] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 783.604968] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 794.215800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 801.203301] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 802.288249] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 813.001765] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-08-3117h43m41s [ 813.473640] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 814.478251] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.18@tcp (stopping) [ 815.075380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 815.076939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 815.081815] Lustre: Skipped 2 previous similar messages [ 815.084942] Lustre: Skipped 1 previous similar message [ 816.097469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 816.099854] Lustre: Skipped 4 previous similar messages [ 820.285128] LustreError: 50566:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 820.309298] Lustre: server umount lustre-MDT0000 complete [ 821.438938] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 821.638937] LustreError: 50080:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 821.647951] LustreError: 50080:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 822.244176] LustreError: 47981:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 822.251238] LustreError: 47981:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 823.655812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 823.692064] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 823.778237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 823.805734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 824.805368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 824.981414] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 826.114592] Lustre: DEBUG MARKER: Wait for df (35967) ... [ 826.587243] Lustre: DEBUG MARKER: done [ 828.896511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 828.898607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 828.902850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 829.408050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 833.646590] LustreError: 51614:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 833.649791] LustreError: 51614:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 833.721584] Lustre: server umount lustre-OST0000 complete [ 835.040795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 835.040956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 835.041843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 835.041848] Lustre: Skipped 2 previous similar messages [ 835.050977] Lustre: Skipped 2 previous similar messages [ 847.328577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 847.330610] Lustre: Skipped 6 previous similar messages [ 849.375133] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 849.406591] LustreError: 51819:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 849.408550] LustreError: 51819:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 849.438953] Lustre: server umount lustre-MDT0000 complete [ 850.403703] LustreError: 47981:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 850.412373] LustreError: 47981:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 850.676521] LustreError: 50298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676660 with bad export cookie 12365340179076523095 [ 850.680327] LustreError: 50298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 850.681582] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 850.837535] Lustre: server umount lustre-MDT0001 complete [ 854.577876] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 855.696168] Key type lgssc unregistered [ 855.835536] LNet: 52552:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.837811] LNetError: 52552:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.846462] LNet: Removed LNI 192.168.201.118@tcp [ 856.175325] Key type .llcrypt unregistered [ 856.176522] Key type ._llcrypt unregistered [ 867.315110] Key type ._llcrypt registered [ 867.316131] Key type .llcrypt registered [ 867.354104] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 873.178296] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 873.606074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 873.612159] alg: No test for adler32 (adler32-zlib) [ 874.460627] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 874.539684] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 876.119133] Key type lgssc registered [ 876.503201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 879.458555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 881.675356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 883.606061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 885.560058] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 887.777465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 887.795956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 887.801042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.879089] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 888.890093] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 888.926854] Lustre: lustre-MDT0000: new disk, initializing [ 888.950911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 888.960434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 890.299588] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 894.441265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 894.469517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 894.506854] Lustre: 56474:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 894.531957] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 894.533768] Lustre: Skipped 1 previous similar message [ 894.573883] Lustre: lustre-MDT0001: new disk, initializing [ 894.607472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 894.618521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 894.621271] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 895.978426] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 899.163463] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 900.417840] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 902.639804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 902.667332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 902.769650] Lustre: lustre-OST0000: new disk, initializing [ 902.771570] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 902.799915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 904.691610] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 906.232619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 906.236181] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 906.266499] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 907.859804] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 909.347109] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 909.420674] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 910.839617] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 910.935406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 913.644918] LustreError: 59414:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 913.707745] Lustre: server umount lustre-OST0000 complete [ 915.424123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 915.427965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 915.434344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 916.448529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 916.448932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 921.568321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 921.571959] Lustre: Skipped 2 previous similar messages [ 926.688596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 926.691127] Lustre: Skipped 1 previous similar message [ 929.247103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 929.282459] LustreError: 59614:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 929.285505] LustreError: 59614:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 929.318142] Lustre: server umount lustre-MDT0000 complete [ 930.533476] LustreError: 56465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676740 with bad export cookie 2680045949796827756 [ 930.536953] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 930.538735] LustreError: 56465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 941.026757] LustreError: 59817:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 941.028396] LustreError: 59817:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 941.102462] Lustre: server umount lustre-MDT0001 complete [ 943.964166] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 17:45:52 (1756676752) [ 946.294273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 946.482241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 947.759260] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 950.596633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 950.731345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 952.070881] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 953.799112] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 955.054695] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 957.349215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 959.416909] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 961.207636] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 979.108770] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-08-3117h46m27s [ 979.615255] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 981.901914] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 982.704630] Lustre: DEBUG MARKER: Creating a test file and stat it [ 982.738701] LustreError: 61837:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1002.898387] Lustre: lustre-MDT0000: Client 8d3c6cbd-6117-4307-aa5b-001fec902399 (at 192.168.201.18@tcp) reconnecting [ 1022.783136] LustreError: 61837:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1023.571549] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1024.444479] Lustre: DEBUG MARKER: done [ 1027.040288] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1027.043129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1027.048710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1028.064629] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1032.108737] LustreError: 63511:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1032.112338] LustreError: 63511:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1032.181694] Lustre: server umount lustre-OST0000 complete [ 1037.791975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1037.794071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1037.796971] Lustre: Skipped 1 previous similar message [ 1037.798659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1037.800301] Lustre: Skipped 2 previous similar messages [ 1039.549711] LustreError: 63712:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1039.552604] LustreError: 63712:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1039.583394] Lustre: server umount lustre-MDT0000 complete [ 1040.794188] LustreError: 60467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676850 with bad export cookie 2680045949796828624 [ 1040.796607] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1040.797584] LustreError: 60467:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1040.944503] Lustre: server umount lustre-MDT0001 complete [ 1044.788199] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1045.842990] Key type lgssc unregistered [ 1045.976350] LNet: 64447:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1045.978380] LNetError: 64447:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1045.984344] LNet: Removed LNI 192.168.201.118@tcp [ 1046.302537] Key type .llcrypt unregistered [ 1046.303775] Key type ._llcrypt unregistered [ 1057.044547] Key type ._llcrypt registered [ 1057.045992] Key type .llcrypt registered [ 1057.085238] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 1063.527692] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1064.016355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1064.028138] alg: No test for adler32 (adler32-zlib) [ 1064.878745] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1064.969286] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1066.559139] Key type lgssc registered [ 1066.927782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1069.955843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1072.188732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1074.294050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1076.529970] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1078.852382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1078.876208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1078.881531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1079.967377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1079.979108] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1080.013455] Lustre: lustre-MDT0000: new disk, initializing [ 1080.037930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1080.046469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1081.359591] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1085.557925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1085.585343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1085.617722] Lustre: 68379:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1085.642540] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1085.645757] Lustre: Skipped 1 previous similar message [ 1085.688362] Lustre: lustre-MDT0001: new disk, initializing [ 1085.724488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1085.739384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1085.743360] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1087.066409] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1090.219132] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1091.559907] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1093.835901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1093.863070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1093.955916] Lustre: lustre-OST0000: new disk, initializing [ 1093.958549] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1093.983760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1094.778602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1094.783697] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1094.818262] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1095.771910] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1098.766145] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1100.088605] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1100.162866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1101.479448] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1101.554381] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1105.380370] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1105.380881] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1105.387642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1105.388603] LustreError: Skipped 1 previous similar message [ 1108.269208] LustreError: 71312:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1108.333819] Lustre: server umount lustre-OST0000 complete [ 1110.496712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1110.497559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1110.502368] Lustre: Skipped 2 previous similar messages [ 1110.504446] Lustre: Skipped 1 previous similar message [ 1115.616268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1115.619179] Lustre: Skipped 2 previous similar messages [ 1115.645033] LustreError: 71513:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1115.647042] LustreError: 71513:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1115.676857] Lustre: server umount lustre-MDT0000 complete [ 1116.963270] LustreError: 69158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676926 with bad export cookie 8299644978875676225 [ 1116.966222] LustreError: 69158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1116.966247] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1117.035552] LustreError: 71715:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1117.039221] LustreError: 71715:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1117.121819] Lustre: server umount lustre-MDT0001 complete [ 1119.802312] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 17:48:48 (1756676928) [ 1120.264902] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1120.765792] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 17:48:49 (1756676929) [ 1124.189343] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1125.484756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1125.630274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1125.964216] Lustre: Failing over lustre-MDT0000 [ 1125.966183] LustreError: 72789:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1125.969718] LustreError: 72789:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1125.971913] LustreError: 72789:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -5 [ 1126.032591] LustreError: 72840:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1126.034923] LustreError: 72840:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1126.103797] Lustre: server umount lustre-MDT0000 complete [ 1128.080454] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 17:48:56 (1756676936) [ 1130.201072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1130.360320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1131.603232] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1134.416482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1135.841152] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1137.581544] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1138.809951] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1141.004772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1141.110167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1141.113840] Lustre: Skipped 1 previous similar message [ 1143.033274] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1144.842882] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1157.549416] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1160.160215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1160.163553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1160.168545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1165.380124] LustreError: 75776:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1165.383477] LustreError: 75776:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1165.413731] Lustre: server umount lustre-MDT0000 complete [ 1165.792626] LustreError: 74132:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1165.797888] LustreError: 74132:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 5 previous similar messages [ 1166.646224] LustreError: 74126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756676976 with bad export cookie 8299644978875677366 [ 1166.649662] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1166.650143] LustreError: 74126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1166.790190] Lustre: server umount lustre-MDT0001 complete [ 1167.886815] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1169.406952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1172.288505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1177.055836] LustreError: 66592:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000744ca09f x1842009209889792/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1177.172629] LustreError: 75015:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1177.205310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1177.240591] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1178.512360] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1181.271485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1181.363229] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1181.368393] Lustre: Skipped 2 previous similar messages [ 1182.654657] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1184.418646] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1185.617992] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1186.527143] Lustre: 66594:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756676980/real 1756676980] req@0000000088aa7fcd x1842009209889408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1756676996 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1186.789467] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1187.807199] Lustre: 66596:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756676981/real 1756676981] req@000000007020ae1c x1842009209889536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1756676997 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1190.883304] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1191.903098] Lustre: 66596:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756676985/real 1756676985] req@000000004be123a3 x1842009209889664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1756677001 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1196.002205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.003168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1196.006150] Lustre: Skipped 2 previous similar messages [ 1196.009838] Lustre: Skipped 4 previous similar messages [ 1198.398290] LustreError: 78238:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1198.399953] LustreError: 78238:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1198.423142] Lustre: server umount lustre-MDT0000 complete [ 1199.632149] LustreError: 73386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677009 with bad export cookie 8299644978875679256 [ 1199.635160] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1199.637431] LustreError: 73386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1199.777521] Lustre: server umount lustre-MDT0001 complete [ 1201.867762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1204.968967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1210.271565] LustreError: 66592:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009a56d411 x1842009209909376/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1210.382047] LustreError: 75015:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1210.403586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1210.405314] Lustre: Skipped 1 previous similar message [ 1210.422236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1211.699460] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1214.437084] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1214.530302] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1215.587363] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1215.856132] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1216.607153] Lustre: 66593:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756677010/real 1756677010] req@00000000b924db23 x1842009209908992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1756677026 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1216.613376] Lustre: 66593:0:(client.c:2470:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1217.544564] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1218.666522] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1221.091307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1221.093732] Lustre: Skipped 2 previous similar messages [ 1222.111144] Lustre: 66595:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756677015/real 1756677015] req@000000004717569d x1842009209909248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1756677031 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1222.119383] Lustre: 66595:0:(client.c:2470:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1226.428227] LustreError: 80690:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1226.430501] LustreError: 80690:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1226.455932] Lustre: server umount lustre-MDT0000 complete [ 1227.685081] LustreError: 78152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677037 with bad export cookie 8299644978875680628 [ 1227.688200] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1227.691154] LustreError: 78152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1227.823973] Lustre: server umount lustre-MDT0001 complete [ 1229.975281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1231.096325] Lustre: DEBUG MARKER: files compared the same [ 1242.102501] Lustre: server umount lustre-OST0000 complete [ 1247.399392] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1248.412274] Key type lgssc unregistered [ 1248.536394] LNet: 82321:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1248.538292] LNetError: 82321:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1248.544282] LNet: Removed LNI 192.168.201.118@tcp [ 1248.847254] Key type .llcrypt unregistered [ 1248.848527] Key type ._llcrypt unregistered [ 1257.476671] Key type ._llcrypt registered [ 1257.478197] Key type .llcrypt registered [ 1257.517868] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1257.969492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1257.987042] alg: No test for adler32 (adler32-zlib) [ 1258.840299] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1258.930955] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1260.511185] Key type lgssc registered [ 1260.897510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1263.560851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1263.568239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1264.725222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1265.991106] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug +malloc all [ 1268.832488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1268.928889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1270.296428] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug +malloc all [ 1272.089483] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1273.283272] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1275.445117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1275.537061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1277.376507] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug +malloc all [ 1279.069667] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1290.915169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1299.424056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1299.428572] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1299.435934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1301.472236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1301.473229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1301.473689] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1301.484867] Lustre: Skipped 1 previous similar message [ 1303.212077] LustreError: 85962:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1303.265172] Lustre: server umount lustre-OST0000 complete [ 1305.055690] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1305.058524] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1305.062584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1310.587960] LustreError: 86164:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1310.590589] LustreError: 86164:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1310.612737] Lustre: server umount lustre-MDT0000 complete [ 1311.711733] LustreError: 84418:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1311.715928] LustreError: 84418:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 1311.949633] LustreError: 84412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677121 with bad export cookie 10742392963598228556 [ 1311.951100] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1311.956027] LustreError: 84412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1312.041383] LustreError: 86367:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1312.043533] LustreError: 86367:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1312.119099] Lustre: server umount lustre-MDT0001 complete [ 1315.788073] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1316.867414] Key type lgssc unregistered [ 1317.005432] LNet: 86898:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1317.010351] LNetError: 86898:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1317.021364] LNet: Removed LNI 192.168.201.118@tcp [ 1317.315249] Key type .llcrypt unregistered [ 1317.316222] Key type ._llcrypt unregistered [ 1326.236920] Key type ._llcrypt registered [ 1326.239238] Key type .llcrypt registered [ 1326.279512] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1326.600262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1326.632556] alg: No test for adler32 (adler32-zlib) [ 1327.480247] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1327.555814] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1329.135194] Key type lgssc registered [ 1329.571896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1332.403595] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1332.412489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1349.919245] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1355.039280] LustreError: 88244:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.118@tcp: failed processing log, type 4: rc = -110 [ 1385.759202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1387.566744] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1389.343370] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1391.885097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1391.921191] Lustre: *** cfs_fail_loc=706, val=0*** [ 1392.055536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1392.075240] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1393.383858] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1396.246416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1396.361633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1397.742450] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1399.540771] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1400.723734] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1401.824867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1401.827072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1401.831136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1402.336200] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1402.338146] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1402.342078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1404.897607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1407.530221] LustreError: 90569:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1407.589891] Lustre: server umount lustre-OST0000 complete [ 1410.018772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1410.020442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1410.026307] Lustre: Skipped 1 previous similar message [ 1410.029371] Lustre: Skipped 1 previous similar message [ 1415.136727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1415.139128] Lustre: Skipped 1 previous similar message [ 1423.327117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1423.364531] LustreError: 90771:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1423.368138] LustreError: 90771:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1423.394940] Lustre: server umount lustre-MDT0000 complete [ 1424.599090] LustreError: 88252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677234 with bad export cookie 1837083537700540194 [ 1424.602395] LustreError: 88252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1424.602522] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1424.661432] LustreError: 90974:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1424.663048] LustreError: 90974:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1424.749454] Lustre: server umount lustre-MDT0001 complete [ 1428.238346] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1429.383030] Key type lgssc unregistered [ 1429.518451] LNet: 91505:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1429.521107] LNetError: 91505:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1429.528411] LNet: Removed LNI 192.168.201.118@tcp [ 1429.858434] Key type .llcrypt unregistered [ 1429.860240] Key type ._llcrypt unregistered [ 1438.436624] Key type ._llcrypt registered [ 1438.437533] Key type .llcrypt registered [ 1438.469371] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1438.753345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1438.758698] alg: No test for adler32 (adler32-zlib) [ 1439.600703] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1439.676987] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1441.255151] Key type lgssc registered [ 1441.592637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1444.080797] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1444.087041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1446.504102] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1449.085845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1449.266674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1450.533410] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1453.243035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1453.349707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1455.179160] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1457.897300] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1457.901956] Lustre: MGS: Client 228713d9-e3b7-46eb-858f-10ff5b296d32 (at 0@lo) reconnecting [ 1457.983851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1458.018017] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1459.181449] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1468.384082] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1468.387241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1468.394431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1469.409335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1469.411099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1469.411825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1473.007132] LustreError: 95509:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1473.072448] Lustre: server umount lustre-OST0000 complete [ 1474.528525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1474.528788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1474.529272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1474.529278] Lustre: Skipped 1 previous similar message [ 1474.537620] Lustre: Skipped 1 previous similar message [ 1479.648698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1479.651954] Lustre: Skipped 2 previous similar messages [ 1480.381881] LustreError: 95713:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1480.383919] LustreError: 95713:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1480.393967] Lustre: server umount lustre-MDT0000 complete [ 1481.691174] LustreError: 95916:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1481.692867] LustreError: 95916:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1481.760571] Lustre: server umount lustre-MDT0001 complete [ 1482.985020] Lustre: server umount MGS complete [ 1486.482115] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 17:54:55 (1756677295) [ 1495.715407] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 1498.621745] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1501.841531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1504.021786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1505.968238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1507.995630] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1510.263369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1510.284931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1511.495136] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1515.329418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1515.351035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1515.424845] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1515.440788] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1515.482389] Lustre: lustre-MDT0001: new disk, initializing [ 1515.508178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1515.517939] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1515.521809] Lustre: 100623:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1515.526689] LustreError: 100623:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1516.771531] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1520.801838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1520.819357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1520.906195] Lustre: lustre-OST0000: new disk, initializing [ 1520.907796] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1520.931822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1522.780560] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1524.896267] Lustre: 101815:0:(client.c:1628:after_reply()) @@@ resending request on EINPROGRESS req@00000000e910cecf x1842009603080704/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1756677350 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1525.843739] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1528.047793] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1528.051344] Lustre: MGS: Client 7289ca2a-44a7-4f64-9a86-87782593d893 (at 0@lo) reconnecting [ 1528.064265] Lustre: 99907:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1528.071324] LustreError: 99907:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1528.080803] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1528.115140] Lustre: lustre-MDT0000: new disk, initializing [ 1528.165417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1529.186086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1529.190755] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1529.255639] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1529.492786] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1548.768049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1548.770656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1548.775121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1554.093209] LustreError: 102858:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1554.095127] LustreError: 102858:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1554.156652] Lustre: server umount lustre-OST0000 complete [ 1560.032549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1560.032768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1560.035857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1560.038370] Lustre: Skipped 2 previous similar messages [ 1560.043161] Lustre: Skipped 3 previous similar messages [ 1561.597460] LustreError: 103061:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1561.600729] LustreError: 103061:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1561.610751] Lustre: server umount lustre-MDT0000 complete [ 1569.031940] Lustre: server umount lustre-MDT0001 complete [ 1570.309361] LustreError: 103464:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1570.312418] LustreError: 103464:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1570.335475] Lustre: server umount MGS complete [ 1574.088755] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 17:56:22 (1756677382) [ 1580.223339] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1581.382813] Key type lgssc unregistered [ 1581.528389] LNet: 104797:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1581.530867] LNetError: 104797:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1581.542326] LNet: Removed LNI 192.168.201.118@tcp [ 1581.873587] Key type .llcrypt unregistered [ 1581.875108] Key type ._llcrypt unregistered [ 1588.693451] Key type ._llcrypt registered [ 1588.694359] Key type .llcrypt registered [ 1588.734150] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1589.070196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1589.100069] alg: No test for adler32 (adler32-zlib) [ 1589.961572] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1590.044570] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1591.623165] Key type lgssc registered [ 1591.985164] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1594.658428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1594.665884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1595.816423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1597.175177] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1599.993698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1600.116929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1601.420951] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1603.179912] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1604.446916] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1606.709598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1606.817885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1608.639737] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1610.365824] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1622.182481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1630.688703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1630.693050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1630.699892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1632.739640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1632.740428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1632.741327] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1634.603072] LustreError: 108246:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1634.666344] Lustre: server umount lustre-OST0000 complete [ 1636.319824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1636.322190] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1636.325842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1636.327788] Lustre: Skipped 1 previous similar message [ 1642.045768] LustreError: 108448:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1642.047980] LustreError: 108448:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1642.080924] Lustre: server umount lustre-MDT0000 complete [ 1642.976130] LustreError: 105977:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1642.980214] LustreError: 105977:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 1643.309215] LustreError: 105956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677453 with bad export cookie 14666286367901293485 [ 1643.311718] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1643.314223] LustreError: 105956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1643.380533] LustreError: 108650:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1643.383391] LustreError: 108650:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1643.460197] Lustre: server umount lustre-MDT0001 complete [ 1647.069079] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1648.104358] Key type lgssc unregistered [ 1648.243461] LNet: 109181:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1648.247254] LNetError: 109181:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1648.255296] LNet: Removed LNI 192.168.201.118@tcp [ 1648.562166] Key type .llcrypt unregistered [ 1648.563222] Key type ._llcrypt unregistered [ 1654.959043] Key type ._llcrypt registered [ 1654.960286] Key type .llcrypt registered [ 1654.998069] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1655.303214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1655.324071] alg: No test for adler32 (adler32-zlib) [ 1656.181752] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1656.264536] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1657.855127] Key type lgssc registered [ 1658.192929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1660.219367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1660.220235] LustreError: 110227:0:(tgt_mount.c:2416:server_fill_super()) cfs_race id 716 sleeping [ 1660.221288] Lustre: Skipped 1 previous similar message [ 1660.224432] LustreError: 110226:0:(tgt_mount.c:2416:server_fill_super()) cfs_fail_race id 716 waking [ 1660.226666] LustreError: 110227:0:(tgt_mount.c:2416:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 1660.228827] LustreError: 110227:0:(tgt_mount.c:2428:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1660.232352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1660.232742] LustreError: 110227:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1661.396721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1663.880449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1663.984552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1665.314792] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1666.862101] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing lsmod [ 1667.846969] LustreError: 111401:0:(tgt_mount.c:2416:server_fill_super()) cfs_race id 716 sleeping [ 1667.847698] LustreError: 111400:0:(tgt_mount.c:2416:server_fill_super()) cfs_fail_race id 716 waking [ 1667.853305] LustreError: 111401:0:(tgt_mount.c:2416:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1667.855692] LustreError: 111401:0:(tgt_mount.c:2428:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1667.859163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1667.859857] LustreError: 111401:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1668.019757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1669.087896] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1669.089886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1669.094078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1673.184653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1673.185705] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1673.186858] Lustre: Skipped 1 previous similar message [ 1673.194442] Lustre: Skipped 1 previous similar message [ 1674.209905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1674.212047] Lustre: Skipped 3 previous similar messages [ 1675.070167] LustreError: 111589:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1675.101291] Lustre: server umount lustre-MDT0000 complete [ 1676.321141] LustreError: 111591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677486 with bad export cookie 10446353423092006289 [ 1676.324029] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1676.324828] LustreError: 111591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1678.303222] LustreError: 110698:0:(client.c:1371:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c4d0cce3 x1842009829555328/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1678.311627] LustreError: 111795:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1678.313249] LustreError: 111795:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1678.380640] Lustre: server umount lustre-MDT0001 complete [ 1689.844826] LustreError: 111995:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1689.846390] LustreError: 111995:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1689.909841] Lustre: server umount lustre-OST0000 complete [ 1692.787625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1692.940828] LustreError: 112473:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1692.946244] LustreError: 112473:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 1692.963993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1694.285767] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1697.265096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1697.398212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1698.825593] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1700.616840] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1701.781472] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1704.133171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1706.054879] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1707.809428] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1717.542313] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1722.850947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1722.851816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1722.852746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1722.856506] Lustre: Skipped 1 previous similar message [ 1725.549205] LustreError: 114739:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1725.551306] LustreError: 114739:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1725.609601] Lustre: server umount lustre-OST0000 complete [ 1727.969950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1727.970857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1727.973968] Lustre: Skipped 1 previous similar message [ 1727.976162] Lustre: Skipped 1 previous similar message [ 1738.208847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1738.211126] Lustre: Skipped 3 previous similar messages [ 1741.279156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1741.315054] LustreError: 114941:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1741.317403] LustreError: 114941:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1741.342206] Lustre: server umount lustre-MDT0000 complete [ 1742.304533] LustreError: 112472:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1742.310384] LustreError: 112472:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 1742.569276] LustreError: 112454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677552 with bad export cookie 10446353423092006947 [ 1742.571369] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1742.573705] LustreError: 112454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1742.724573] Lustre: server umount lustre-MDT0001 complete [ 1746.373025] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1747.484664] Key type lgssc unregistered [ 1747.614368] LNet: 115675:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1747.616767] LNetError: 115675:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1747.623321] LNet: Removed LNI 192.168.201.118@tcp [ 1747.931388] Key type .llcrypt unregistered [ 1747.933279] Key type ._llcrypt unregistered [ 1756.540440] Key type ._llcrypt registered [ 1756.541403] Key type .llcrypt registered [ 1756.581521] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1756.920642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1756.975669] alg: No test for adler32 (adler32-zlib) [ 1757.834324] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1757.929526] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1759.511203] Key type lgssc registered [ 1759.851990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1762.388270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1762.396728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1763.544424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1764.814839] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1767.544434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1767.661332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1768.948650] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1770.711080] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1771.889792] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1774.014020] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1774.049712] LustreError: 118646:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1774.053579] LustreError: 118646:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 1774.120795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1775.842875] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1777.624608] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1778.221594] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 1785.887415] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1786.642603] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1788.390454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1788.392527] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1788.393444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1788.401253] Lustre: Skipped 1 previous similar message [ 1788.405918] LustreError: Skipped 1 previous similar message [ 1793.506399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1793.509127] Lustre: Skipped 2 previous similar messages [ 1793.834879] LustreError: 119395:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1793.893068] Lustre: server umount lustre-OST0000 complete [ 1798.625083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1798.625254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1798.625891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1798.633934] Lustre: Skipped 1 previous similar message [ 1801.530974] LustreError: 119600:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1801.534208] LustreError: 119600:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1801.564401] Lustre: server umount lustre-MDT0000 complete [ 1802.823191] LustreError: 119248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677612 with bad export cookie 4824294490324547454 [ 1802.826105] LustreError: 119248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1802.826745] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1802.888019] LustreError: 119802:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1802.891040] LustreError: 119802:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1802.968711] Lustre: server umount lustre-MDT0001 complete [ 1806.509922] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1807.623170] Key type lgssc unregistered [ 1807.749412] LNet: 120334:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1807.751611] LNetError: 120334:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1807.761432] LNet: Removed LNI 192.168.201.118@tcp [ 1808.093125] Key type .llcrypt unregistered [ 1808.094192] Key type ._llcrypt unregistered [ 1815.127810] Key type ._llcrypt registered [ 1815.129702] Key type .llcrypt registered [ 1815.170452] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1815.501218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1815.523469] alg: No test for adler32 (adler32-zlib) [ 1816.372759] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1816.453543] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1818.031176] Key type lgssc registered [ 1818.390167] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1820.930522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1820.936962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1822.082769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1823.372383] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1826.117929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1826.240228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1827.558272] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1829.328836] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1830.513585] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1832.776453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1832.893111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1834.772826] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1836.534226] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1847.271177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 1856.479832] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1856.482033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1856.485906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1857.505063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1857.505563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1857.508083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1860.910114] LustreError: 123810:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1860.970513] Lustre: server umount lustre-OST0000 complete [ 1862.624087] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1862.629513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1862.631974] Lustre: Skipped 1 previous similar message [ 1867.231883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1867.234918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1867.238800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1868.285028] LustreError: 124012:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1868.287274] LustreError: 124012:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1868.314694] Lustre: server umount lustre-MDT0000 complete [ 1869.552284] LustreError: 123145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677679 with bad export cookie 18183188972252122834 [ 1869.555850] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1869.557720] LustreError: 123145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1869.627581] LustreError: 124213:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1869.629755] LustreError: 124213:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1869.718747] Lustre: server umount lustre-MDT0001 complete [ 1873.505972] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1874.547098] Key type lgssc unregistered [ 1874.677398] LNet: 124743:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1874.679504] LNetError: 124743:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1874.689338] LNet: Removed LNI 192.168.201.118@tcp [ 1874.987432] Key type .llcrypt unregistered [ 1874.988656] Key type ._llcrypt unregistered [ 1883.561374] Key type ._llcrypt registered [ 1883.562312] Key type .llcrypt registered [ 1883.599609] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1883.914936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1883.947061] alg: No test for adler32 (adler32-zlib) [ 1884.802336] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1884.882822] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1886.463135] Key type lgssc registered [ 1886.835284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1889.684036] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1889.692237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1890.844070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1892.104253] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1894.908952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1895.035905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1896.363157] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1898.106738] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1899.252101] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1901.407424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1901.521792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1903.309890] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1904.048121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 1904.977123] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1908.329067] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1909.532772] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1909.536222] Lustre: Skipped 2 previous similar messages [ 1910.914678] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1910.917274] Lustre: Skipped 2 previous similar messages [ 1911.233188] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1911.247797] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1922.724559] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.18@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1922.728062] Lustre: Skipped 2 previous similar messages [ 1929.208754] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.18@tcp 192.168.0.[2,10]@tcp [ 1930.551067] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.18/24@tcp in log lustre-MDT0000 [ 1930.553395] Lustre: Skipped 2 previous similar messages [ 1935.350487] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.18/24@tcp [ 1935.352384] Lustre: Skipped 1 previous similar message [ 1939.935977] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1939.939349] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1939.943719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1940.965914] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1940.966548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1940.968158] LustreError: Skipped 5 previous similar messages [ 1940.972232] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1944.365131] LustreError: 129836:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1944.425836] Lustre: server umount lustre-OST0000 complete [ 1946.081531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.082276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1946.088674] Lustre: Skipped 1 previous similar message [ 1946.091110] Lustre: Skipped 1 previous similar message [ 1951.200421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1951.204734] Lustre: Skipped 2 previous similar messages [ 1951.871280] LustreError: 130038:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1951.873206] LustreError: 130038:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1951.906372] Lustre: server umount lustre-MDT0000 complete [ 1953.170592] LustreError: 128411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677762 with bad export cookie 6830322718460051394 [ 1953.171653] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1953.176343] LustreError: 128411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1953.239460] LustreError: 130240:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1953.241102] LustreError: 130240:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1953.351466] Lustre: server umount lustre-MDT0001 complete [ 1957.054022] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 1958.104152] Key type lgssc unregistered [ 1958.248444] LNet: 130771:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1958.251300] LNetError: 130771:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1958.259368] LNet: Removed LNI 192.168.201.118@tcp [ 1958.611577] Key type .llcrypt unregistered [ 1958.612572] Key type ._llcrypt unregistered [ 1968.727003] Key type ._llcrypt registered [ 1968.728165] Key type .llcrypt registered [ 1968.771423] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 1969.083029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1969.110985] alg: No test for adler32 (adler32-zlib) [ 1969.963681] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 1970.046833] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 1971.623170] Key type lgssc registered [ 1971.967316] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1974.355589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1974.362602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1975.517672] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1975.523203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1975.526432] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.18/24@tcp [ 1976.731936] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1979.559417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1979.664785] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1979.673411] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.18/24@tcp [ 1979.693546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1980.934418] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1982.723189] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1983.890109] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1986.097680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1986.213108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1987.973982] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 1989.805298] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2000.553187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2004.960842] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2004.965702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2004.973577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2005.985041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2005.985856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2005.990675] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2006.316696] LustreError: 134615:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2006.378765] Lustre: server umount lustre-OST0000 complete [ 2009.722570] LustreError: 134816:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2009.724989] LustreError: 134816:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2009.751445] Lustre: server umount lustre-MDT0000 complete [ 2010.591909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2010.593872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2010.597478] LustreError: 132289:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2010.603746] LustreError: 132289:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2010.989378] LustreError: 132269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677820 with bad export cookie 9982302627557423183 [ 2010.993638] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2010.994465] LustreError: 132269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2011.061966] LustreError: 135019:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2011.063737] LustreError: 135019:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2011.141848] Lustre: server umount lustre-MDT0001 complete [ 2014.997038] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2016.173308] Key type lgssc unregistered [ 2016.321481] LNet: 135550:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2016.323496] LNetError: 135550:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2016.330410] LNet: Removed LNI 192.168.201.118@tcp [ 2016.683490] Key type .llcrypt unregistered [ 2016.684914] Key type ._llcrypt unregistered [ 2025.004418] Key type ._llcrypt registered [ 2025.005419] Key type .llcrypt registered [ 2025.045257] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 2031.130162] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2031.527187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2031.534463] alg: No test for adler32 (adler32-zlib) [ 2032.413280] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 2032.500693] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2034.079151] Key type lgssc registered [ 2034.428805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2037.325949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2039.517816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2041.495553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2043.477054] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2045.686990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2045.707911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2045.712766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2046.801493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2046.816595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2046.857525] Lustre: lustre-MDT0000: new disk, initializing [ 2046.882647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2046.889771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2048.180751] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2052.494686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2052.526639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2052.570535] Lustre: 138896:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2052.602427] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2052.606118] Lustre: Skipped 1 previous similar message [ 2052.653877] Lustre: lustre-MDT0001: new disk, initializing [ 2052.698819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2052.712981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2052.716747] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2054.207598] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2058.045866] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2059.459508] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2061.698239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2061.728308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2061.816661] Lustre: lustre-OST0000: new disk, initializing [ 2061.818350] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2061.846849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2063.681842] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2064.829794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2064.832664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2064.866878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2066.786223] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2068.207870] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2068.292283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2069.555646] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2069.623931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2075.106833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2075.106953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2075.109667] LustreError: Skipped 1 previous similar message [ 2075.113247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2075.116637] Lustre: Skipped 1 previous similar message [ 2076.333108] LustreError: 141837:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2076.402452] Lustre: server umount lustre-OST0000 complete [ 2078.176863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2078.182771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2078.191901] Lustre: Skipped 1 previous similar message [ 2078.195995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2080.224708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2080.225127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2083.962900] LustreError: 142038:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2083.965617] LustreError: 142038:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2083.994195] Lustre: server umount lustre-MDT0000 complete [ 2085.187432] LustreError: 140643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677894 with bad export cookie 13153461122999338224 [ 2085.191019] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2085.194936] LustreError: 140643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2085.254067] LustreError: 142240:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2085.255804] LustreError: 142240:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2085.353223] Lustre: server umount lustre-MDT0001 complete [ 2088.017652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2088.192874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2089.445659] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2092.135394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2092.243422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2093.456938] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2095.142485] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2096.265531] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2098.348571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2100.255280] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2101.942533] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2103.226970] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2113.522213] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 2114.753705] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2114.826053] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2118.177385] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2118.198617] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2118.251927] Lustre: lustre-OST0001: new disk, initializing [ 2118.254401] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2118.284751] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2118.287390] Lustre: Skipped 1 previous similar message [ 2120.126228] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2123.270048] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2125.820991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2125.827531] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2125.857367] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2126.405333] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2127.611815] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2127.680655] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2128.806086] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff914e86925800.ost_server_uuid 50 [ 2129.291849] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff914e86925800.ost_server_uuid in FULL state after 0 sec [ 2136.036468] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2136.036644] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2136.038572] LustreError: Skipped 1 previous similar message [ 2136.045361] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2136.047481] Lustre: Skipped 2 previous similar messages [ 2137.198080] LustreError: 147056:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2137.200755] LustreError: 147056:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2137.220103] Lustre: server umount lustre-OST0001 complete [ 2138.592467] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2141.154382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2141.155083] LustreError: 144314:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2141.155393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2141.155400] Lustre: Skipped 2 previous similar messages [ 2141.156332] Lustre: Skipped 1 previous similar message [ 2141.167585] LustreError: 144314:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 2144.556177] LustreError: 147257:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2144.558018] LustreError: 147257:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2144.615187] Lustre: server umount lustre-OST0000 complete [ 2151.392081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2151.393675] Lustre: Skipped 4 previous similar messages [ 2152.029025] Lustre: server umount lustre-MDT0000 complete [ 2153.233143] LustreError: 145791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756677962 with bad export cookie 13153461122999339106 [ 2153.236321] LustreError: 145791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2153.236397] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2153.304669] LustreError: 147661:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2153.306413] LustreError: 147661:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2153.390285] Lustre: server umount lustre-MDT0001 complete [ 2159.681764] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2160.825310] Key type lgssc unregistered [ 2160.948314] LNet: 148648:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2160.951489] LNetError: 148648:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2160.957298] LNet: Removed LNI 192.168.201.118@tcp [ 2161.255929] Key type .llcrypt unregistered [ 2161.257235] Key type ._llcrypt unregistered [ 2171.991238] Key type ._llcrypt registered [ 2171.994343] Key type .llcrypt registered [ 2172.034226] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 2178.036964] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2178.464608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2178.475327] alg: No test for adler32 (adler32-zlib) [ 2179.315335] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 2179.390253] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2180.967206] Key type lgssc registered [ 2181.317681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2184.074546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2186.158406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2188.122947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2190.108651] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2192.229609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2192.252202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2192.257352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.350981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2193.364829] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2193.401603] Lustre: lustre-MDT0000: new disk, initializing [ 2193.424876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2193.431329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2194.659491] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2198.773206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2198.800194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2198.843708] Lustre: 152569:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2198.868899] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2198.871273] Lustre: Skipped 1 previous similar message [ 2198.910234] Lustre: lustre-MDT0001: new disk, initializing [ 2198.946603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2198.960562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2198.963615] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2200.350282] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2203.440397] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2204.602641] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2206.817910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2206.844596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2206.942921] Lustre: lustre-OST0000: new disk, initializing [ 2206.944968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2206.972369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2208.055976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2208.060258] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2208.113433] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2208.790044] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2211.807608] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2213.255727] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2213.350041] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2214.714330] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2214.799558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2217.454112] LustreError: 155503:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2217.519804] Lustre: server umount lustre-OST0000 complete [ 2219.487848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2219.489787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2219.493227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2223.584473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2223.585182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2228.704445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2228.706060] Lustre: Skipped 2 previous similar messages [ 2233.311166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2233.340821] LustreError: 155703:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2233.342651] LustreError: 155703:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2233.368601] Lustre: server umount lustre-MDT0000 complete [ 2234.622697] LustreError: 152560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678044 with bad export cookie 2134812766848483202 [ 2234.625509] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2234.627661] LustreError: 152560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2245.091032] LustreError: 155906:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 2245.092764] LustreError: 155906:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2245.188623] Lustre: server umount lustre-MDT0001 complete [ 2248.050564] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 18:07:36 (1756678056) [ 2253.071600] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 2255.801496] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2258.898724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2261.122660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2263.089921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2265.082988] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2267.246327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2267.272839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2267.351919] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2267.364345] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2267.400973] Lustre: lustre-MDT0000: new disk, initializing [ 2267.426935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2267.433516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2268.731533] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2272.740900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2272.763670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2272.794061] Lustre: 158892:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2272.818529] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2272.821011] Lustre: Skipped 1 previous similar message [ 2272.866090] Lustre: lustre-MDT0001: new disk, initializing [ 2272.902602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2272.916903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2272.919596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2274.185620] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2277.139175] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2278.287681] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2280.409757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2280.431947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2282.419452] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2285.471981] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2285.557403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2285.560238] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2285.592223] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2287.405568] Lustre: Failing over lustre-OST0000 [ 2287.429119] LustreError: 161334:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2287.430945] LustreError: 161334:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2287.505686] Lustre: server umount lustre-OST0000 complete [ 2300.258439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2300.367986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2300.371482] Lustre: Skipped 1 previous similar message [ 2300.374092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2300.377571] Lustre: Skipped 1 previous similar message [ 2300.381522] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2302.141517] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2302.348811] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2303.319286] Lustre: Failing over lustre-MDT0000 [ 2303.382166] LustreError: 162304:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2303.383777] LustreError: 162304:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2303.410237] Lustre: server umount lustre-MDT0000 complete [ 2303.968779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2303.971286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2303.975365] LustreError: 158904:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2303.979599] LustreError: 158904:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 2305.504202] LustreError: 158904:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2305.509796] LustreError: 158904:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2306.399108] Lustre: 150786:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756678100/real 1756678100] req@00000000273e1852 x1842010379046400/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1756678116 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2307.469763] LustreError: 160647:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2310.624109] LustreError: 159687:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2310.627922] LustreError: 159687:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2311.519117] Lustre: 150786:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1756678105/real 1756678105] req@0000000036929c80 x1842010379048448/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1756678121 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2315.744079] LustreError: 159687:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2315.748249] LustreError: 159687:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 2316.115448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2316.153680] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2316.157610] Lustre: 158895:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.118@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2316.225817] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2316.267508] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2317.454592] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2317.703781] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2321.384453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2321.392344] Lustre: lustre-OST0000: Recovery over after 0:19, of 3 clients 3 recovered and 0 were evicted. [ 2321.394753] Lustre: Skipped 1 previous similar message [ 2321.417427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2326.498586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2326.499505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2326.501192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2326.504141] Lustre: Skipped 3 previous similar messages [ 2326.507245] Lustre: Skipped 1 previous similar message [ 2328.426117] LustreError: 163188:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2328.429326] LustreError: 163188:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2328.489715] Lustre: server umount lustre-OST0000 complete [ 2331.617626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2331.617862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2331.619474] LustreError: Skipped 1 previous similar message [ 2331.622615] Lustre: Skipped 1 previous similar message [ 2335.767472] Lustre: server umount lustre-MDT0000 complete [ 2336.736969] LustreError: 159687:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2337.004805] LustreError: 163393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678146 with bad export cookie 2134812766848485197 [ 2337.007777] LustreError: 163393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2337.009094] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2337.181739] Lustre: server umount lustre-MDT0001 complete [ 2340.691652] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2341.746340] Key type lgssc unregistered [ 2341.882355] LNet: 164124:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2341.884384] LNetError: 164124:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2341.894297] LNet: Removed LNI 192.168.201.118@tcp [ 2342.194723] Key type .llcrypt unregistered [ 2342.195812] Key type ._llcrypt unregistered [ 2362.350103] Key type ._llcrypt registered [ 2362.351245] Key type .llcrypt registered [ 2362.385205] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2362.731711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2362.782121] alg: No test for adler32 (adler32-zlib) [ 2363.624498] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 2363.705183] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2365.287155] Key type lgssc registered [ 2365.647587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2368.214137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2368.224039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2369.387548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2370.729976] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2373.887912] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2375.872834] LustreError: 165651:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2375.877666] LustreError: 165651:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2375.880186] LustreError: 165651:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 2375.942513] LustreError: 166175:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2376.024685] Lustre: server umount lustre-MDT0000 complete [ 2380.887866] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 2383.911386] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2387.464080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2389.802821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2391.755567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2393.733140] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2396.012180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2396.035762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2396.120302] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2396.132152] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2396.169061] Lustre: lustre-MDT0000: new disk, initializing [ 2396.194312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2396.200779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2397.494986] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2401.635201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2401.660696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2401.695152] Lustre: 168815:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2401.743190] LustreError: 168825:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2401.748625] LustreError: 168825:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2401.749741] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2401.753952] Lustre: Skipped 1 previous similar message [ 2401.794335] Lustre: lustre-MDT0001: new disk, initializing [ 2401.832503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2401.848902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2401.853211] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2403.153356] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2406.090228] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2407.245863] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2409.532655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2409.557260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2409.687201] Lustre: lustre-OST0000: new disk, initializing [ 2409.689783] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2409.727386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2411.606370] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2414.756155] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2419.191825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2419.194897] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2419.231970] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2421.740730] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2422.275190] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 18:10:31 (1756678231) [ 2488.799489] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2488.801929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2488.805244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2490.847746] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2490.847921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2490.853319] Lustre: Skipped 1 previous similar message [ 2490.855550] Lustre: Skipped 1 previous similar message [ 2492.666913] LustreError: 172317:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2492.668823] LustreError: 172317:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2492.722706] Lustre: server umount lustre-MDT0000 complete [ 2493.943483] LustreError: 168807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678303 with bad export cookie 18170784385745578032 [ 2493.945029] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2493.948121] LustreError: 168807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2495.967620] LustreError: 168821:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2495.967685] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2495.977734] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2495.979591] Lustre: Skipped 1 previous similar message [ 2498.975647] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2498.975647] LustreError: 168821:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2504.095633] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2504.095634] LustreError: 168821:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2508.255127] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2508.285746] LustreError: 172519:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2508.287715] LustreError: 172519:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2508.368186] Lustre: server umount lustre-MDT0001 complete [ 2515.689703] LustreError: 172721:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2515.691490] LustreError: 172721:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2515.755572] Lustre: server umount lustre-OST0000 complete [ 2518.734342] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 2521.718465] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2525.434752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2527.663592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2529.610048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2531.635688] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2533.807049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2533.830295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2533.927150] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2533.939660] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2533.975197] Lustre: lustre-MDT0000: new disk, initializing [ 2533.997820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2534.006182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2535.245139] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2539.210408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2539.232061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2539.260863] Lustre: 175054:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2539.280509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2539.281991] Lustre: Skipped 1 previous similar message [ 2539.316016] Lustre: lustre-MDT0001: new disk, initializing [ 2539.357826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2539.361169] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2540.545765] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2543.558190] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2544.659601] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2546.810961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2546.834755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2546.967346] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2546.970727] Lustre: Skipped 1 previous similar message [ 2548.752357] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2549.558106] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2549.561218] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2549.598884] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2551.891167] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2553.272430] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2553.360790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2554.671735] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2554.743677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2557.419075] LustreError: 177991:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2557.420736] LustreError: 177991:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2557.485106] Lustre: server umount lustre-OST0000 complete [ 2559.969272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2559.970031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2559.972653] Lustre: Skipped 1 previous similar message [ 2559.972839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2559.974713] Lustre: Skipped 1 previous similar message [ 2569.187383] LustreError: 178192:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2569.190843] LustreError: 178192:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2569.215592] Lustre: server umount lustre-MDT0000 complete [ 2570.208720] LustreError: 175065:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2570.213828] LustreError: 175065:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2570.464461] LustreError: 175835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678380 with bad export cookie 18170784385745736855 [ 2570.467255] LustreError: 175835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2570.467564] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2581.040628] Lustre: server umount lustre-MDT0001 complete [ 2583.892623] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:13:12 (1756678392) [ 2589.081479] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 2592.020453] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2595.086565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2597.197573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2599.108238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2601.160319] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2603.436472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2603.460557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2603.557498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2603.573921] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2603.575466] Lustre: Skipped 1 previous similar message [ 2603.613402] Lustre: lustre-MDT0000: new disk, initializing [ 2603.615912] Lustre: Skipped 1 previous similar message [ 2603.643732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2603.651371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2605.009421] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2609.121095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2609.149311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2609.187787] Lustre: 181381:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2609.292070] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2610.513840] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2613.514233] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2614.658269] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2616.814688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2616.843843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2618.727702] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2621.048671] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2621.089638] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2621.686484] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2626.530168] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2626.530546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2626.532754] LustreError: Skipped 1 previous similar message [ 2626.533355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2626.535962] Lustre: Skipped 1 previous similar message [ 2626.537329] Lustre: Skipped 2 previous similar messages [ 2629.549479] LustreError: 183927:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2629.551621] LustreError: 183927:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 2629.624400] Lustre: server umount lustre-OST0000 complete [ 2637.011960] Lustre: server umount lustre-MDT0000 complete [ 2638.255226] LustreError: 181372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678447 with bad export cookie 18170784385745737730 [ 2638.258279] LustreError: 181372:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2638.258417] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2638.414891] Lustre: server umount lustre-MDT0001 complete [ 2641.022758] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:14:09 (1756678449) [ 2646.024035] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 2648.871504] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2652.010939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2654.148302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2656.092097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2658.125866] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2660.417980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2660.447517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2660.551593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2660.566074] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2660.569306] Lustre: Skipped 3 previous similar messages [ 2660.607719] Lustre: lustre-MDT0000: new disk, initializing [ 2660.608950] Lustre: Skipped 2 previous similar messages [ 2660.635309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2660.637145] Lustre: Skipped 2 previous similar messages [ 2660.642841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2660.647688] Lustre: Skipped 2 previous similar messages [ 2662.133127] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2666.239898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2666.263912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2666.297461] Lustre: 187313:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2666.406385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2667.692702] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2670.789541] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2672.032899] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2674.290685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2674.319897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2676.074151] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2676.388703] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2679.478633] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2686.434830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2686.434878] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2686.438976] LustreError: Skipped 1 previous similar message [ 2686.442178] Lustre: Skipped 2 previous similar messages [ 2686.443982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2686.446686] Lustre: Skipped 5 previous similar messages [ 2687.469657] LustreError: 189861:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2687.471494] LustreError: 189861:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2687.533847] Lustre: server umount lustre-OST0000 complete [ 2703.327105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2703.380087] Lustre: server umount lustre-MDT0000 complete [ 2704.574339] LustreError: 187305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678514 with bad export cookie 18170784385745738801 [ 2704.576802] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2704.580988] LustreError: 187305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2704.723687] Lustre: server umount lustre-MDT0001 complete [ 2708.339138] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2709.449234] Key type lgssc unregistered [ 2709.597362] LNet: 190797:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2709.599503] LNetError: 190797:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2709.610274] LNet: Removed LNI 192.168.201.118@tcp [ 2709.921420] Key type .llcrypt unregistered [ 2709.923235] Key type ._llcrypt unregistered [ 2718.542505] Key type ._llcrypt registered [ 2718.544277] Key type .llcrypt registered [ 2718.580703] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2718.932785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2718.961593] alg: No test for adler32 (adler32-zlib) [ 2719.804975] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 2719.883672] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2721.463152] Key type lgssc registered [ 2721.802661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2724.321778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2724.329381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2725.485270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2726.708129] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2729.574478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2729.724829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2731.044865] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2732.747229] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2733.955765] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2736.177453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2736.306601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2738.161246] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2739.933083] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2750.629661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2770.400264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2770.402656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2770.406329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2771.427393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2771.427623] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2771.429339] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2773.038038] LustreError: 194449:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2773.103506] Lustre: server umount lustre-OST0000 complete [ 2776.031859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2776.034046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2776.037849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2776.040603] Lustre: Skipped 1 previous similar message [ 2780.429586] LustreError: 194651:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2780.434170] LustreError: 194651:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2780.470460] Lustre: server umount lustre-MDT0000 complete [ 2781.664992] LustreError: 194450:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2781.669568] LustreError: 194450:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 2781.775314] LustreError: 193798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678591 with bad export cookie 13002492746147219962 [ 2781.778106] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2781.778744] LustreError: 193798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2781.839261] LustreError: 194854:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2781.841223] LustreError: 194854:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2781.929764] Lustre: server umount lustre-MDT0001 complete [ 2785.525592] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 2786.619551] Key type lgssc unregistered [ 2786.754417] LNet: 195386:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2786.756759] LNetError: 195386:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2786.764291] LNet: Removed LNI 192.168.201.118@tcp [ 2787.076813] Key type .llcrypt unregistered [ 2787.077973] Key type ._llcrypt unregistered [ 2796.575582] Key type ._llcrypt registered [ 2796.576713] Key type .llcrypt registered [ 2796.617764] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 2796.922263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2796.928571] alg: No test for adler32 (adler32-zlib) [ 2797.777696] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 2797.864588] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 2799.455185] Key type lgssc registered [ 2799.971574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2802.697397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2802.711051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2803.909802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2805.149224] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2808.047783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2808.174647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2809.431269] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2811.142927] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2812.286216] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2814.503451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2814.619199] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2816.332693] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2817.980317] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2828.965970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2838.496069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2838.499408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2838.503242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2839.521180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2839.521344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2839.521619] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2839.524976] Lustre: Skipped 1 previous similar message [ 2842.092121] LustreError: 199039:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2842.165102] Lustre: server umount lustre-OST0000 complete [ 2844.073670] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff914ea0188800.ost_server_uuid 50 [ 2844.608082] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2854.591429] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2854.598334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2859.488050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2859.491631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2859.497193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2860.736314] LustreError: 199387:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2860.739280] LustreError: 199387:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2860.769973] Lustre: server umount lustre-MDT0000 complete [ 2862.023393] LustreError: 196755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678671 with bad export cookie 10372088357982391368 [ 2862.026268] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2862.028867] LustreError: 196755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2862.085623] LustreError: 199588:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2862.087453] LustreError: 199588:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2862.171613] Lustre: server umount lustre-MDT0001 complete [ 2864.893238] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 18:17:53 (1756678673) [ 2867.076517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.251260] LustreError: 200259:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2867.257827] LustreError: 200259:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2867.280576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2868.680399] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2871.624944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2871.769458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2873.255786] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2875.186368] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2876.465267] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2878.691970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2880.690805] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2882.361237] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2884.516612] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2884.552762] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2884.616919] Lustre: lustre-OST0001: new disk, initializing [ 2884.618922] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2884.655094] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2884.656754] Lustre: Skipped 1 previous similar message [ 2886.434524] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2889.454144] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2891.815853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2892.798865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2892.803694] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2892.828735] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2896.864066] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2896.867464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2896.873395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2896.874906] Lustre: Skipped 1 previous similar message [ 2901.484113] LustreError: 203466:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2901.485800] LustreError: 203466:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2901.501474] Lustre: server umount lustre-OST0000 complete [ 2903.011820] LustreError: 201855:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2903.015825] LustreError: 201855:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2903.384630] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2903.452941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2904.687437] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2904.771821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2904.969022] LustreError: 203373:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2908.131489] LustreError: 201856:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2908.135309] LustreError: 201856:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2913.251804] LustreError: 201857:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2913.256039] LustreError: 201857:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 2918.372204] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2918.372396] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2918.376744] Lustre: Skipped 1 previous similar message [ 2918.378540] LustreError: Skipped 1 previous similar message [ 2918.382084] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2918.385452] Lustre: Skipped 4 previous similar messages [ 2920.813615] LustreError: 204166:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2920.815469] LustreError: 204166:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2920.885577] Lustre: server umount lustre-OST0001 complete [ 2928.212955] Lustre: server umount lustre-MDT0000 complete [ 2928.607947] LustreError: 200277:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2928.612633] LustreError: 200277:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 2929.397138] LustreError: 200240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678739 with bad export cookie 10372088357982392425 [ 2929.399931] LustreError: 200240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2929.400618] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2929.460087] LustreError: 204570:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2929.461620] LustreError: 204570:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2929.539540] Lustre: server umount lustre-MDT0001 complete [ 2931.261779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2932.484326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2933.754357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2934.989617] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2937.099972] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 18:19:05 (1756678745) [ 2939.279570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2939.399888] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2939.413099] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2939.495181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2940.782541] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2943.672750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2943.704405] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2943.723394] Lustre: 206061:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2945.208408] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2947.043112] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2948.319387] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2950.563994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2950.638101] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2952.605763] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2952.845524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 2954.363196] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2956.677312] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2956.701343] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2958.678199] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 2960.506680] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2961.890698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2961.891935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2961.894199] Lustre: Skipped 4 previous similar messages [ 2961.896311] Lustre: Skipped 3 previous similar messages [ 2967.468482] LustreError: 209245:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2967.470360] LustreError: 209245:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2967.483602] Lustre: server umount lustre-OST0000 complete [ 2968.038632] LustreError: 208081:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2968.044175] LustreError: 208081:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 2978.274041] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2978.277655] LustreError: Skipped 2 previous similar messages [ 2984.243759] Lustre: server umount lustre-OST0001 complete [ 2991.639375] Lustre: server umount lustre-MDT0000 complete [ 2992.773379] LustreError: 206053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678802 with bad export cookie 10372088357982393811 [ 2992.776734] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2992.779034] LustreError: 206053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2992.923432] Lustre: server umount lustre-MDT0001 complete [ 2994.680163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2995.904186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2997.143429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2998.389863] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3000.370351] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 18:20:09 (1756678809) [ 3005.469053] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 3008.359633] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3011.981142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3014.145902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3016.131305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3018.069471] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3020.294400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3020.319996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3020.402977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3020.416706] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3020.456706] Lustre: lustre-MDT0000: new disk, initializing [ 3020.480672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3020.482434] Lustre: Skipped 3 previous similar messages [ 3020.487628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3021.689962] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3025.784394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3025.806175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3025.835170] Lustre: 213676:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3025.867835] LustreError: 213687:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3025.875240] LustreError: 213687:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 8 previous similar messages [ 3025.879351] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3025.881424] Lustre: Skipped 1 previous similar message [ 3025.914274] Lustre: lustre-MDT0001: new disk, initializing [ 3025.956560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3025.960376] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3027.213954] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3030.319868] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3031.607746] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3033.946407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3033.973819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3034.083994] Lustre: lustre-OST0000: new disk, initializing [ 3034.085768] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3034.742954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3034.746763] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3034.794862] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3036.074280] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3039.095259] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3040.383348] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3040.452370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3041.761730] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3041.825561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3045.347359] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3045.347752] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3045.350242] LustreError: Skipped 1 previous similar message [ 3045.355393] Lustre: Skipped 4 previous similar messages [ 3045.358707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3045.362224] Lustre: Skipped 11 previous similar messages [ 3048.491784] LustreError: 216612:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3048.493591] LustreError: 216612:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3048.577781] Lustre: server umount lustre-OST0000 complete [ 3064.287081] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3064.348654] Lustre: server umount lustre-MDT0000 complete [ 3065.626834] LustreError: 213666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678875 with bad export cookie 10372088357982395330 [ 3065.629795] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3065.630178] LustreError: 213666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3065.819747] Lustre: server umount lustre-MDT0001 complete [ 3068.695293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3070.171714] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3073.061161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3074.465318] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3076.312866] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3077.498109] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3079.766827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3081.855885] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3083.680326] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3085.105597] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3094.377419] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 3095.703382] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3095.770732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3098.592338] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3102.651477] Lustre: server umount lustre-OST0000 complete [ 3104.679482] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3104.764120] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3106.116408] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3130.607098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3130.715127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3130.718952] Lustre: Skipped 5 previous similar messages [ 3132.800879] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3135.166794] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3135.876841] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3140.129543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3140.133322] Lustre: Skipped 9 previous similar messages [ 3143.145414] LustreError: 221951:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3143.147877] LustreError: 221951:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3143.215315] Lustre: server umount lustre-OST0000 complete [ 3145.188408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3145.197848] Lustre: Skipped 5 previous similar messages [ 3151.134721] Lustre: server umount lustre-MDT0000 complete [ 3152.767895] LustreError: 220303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756678962 with bad export cookie 10372088357982396184 [ 3152.773563] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3152.776139] LustreError: 220303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3152.959741] Lustre: server umount lustre-MDT0001 complete [ 3156.793654] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 18:22:45 (1756678965) [ 3159.932913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3160.099506] LustreError: 223029:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3160.105596] LustreError: 223029:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 3161.521833] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3164.442971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3165.978696] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3167.900302] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3169.145948] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3171.397126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3173.316852] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3175.090562] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3176.490551] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3185.780443] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 3187.163580] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3187.253976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3189.440046] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3189.473197] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3189.529303] Lustre: lustre-OST0001: new disk, initializing [ 3189.531227] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3191.445277] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3194.539701] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3195.921344] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3200.024071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3200.028602] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3200.074801] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3201.118482] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 3202.356246] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3202.434709] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3205.092486] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3205.094550] LustreError: Skipped 3 previous similar messages [ 3209.086800] Lustre: server umount lustre-OST0001 complete [ 3210.962524] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3211.045509] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3212.361442] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3212.449655] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3235.958260] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3237.876018] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3239.703634] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3240.277155] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3247.034131] Lustre: server umount lustre-OST0001 complete [ 3265.286813] Lustre: server umount lustre-OST0000 complete [ 3272.160871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3272.163083] Lustre: Skipped 18 previous similar messages [ 3281.375204] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3281.433436] LustreError: 229405:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3281.437207] LustreError: 229405:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 3281.475114] Lustre: server umount lustre-MDT0000 complete [ 3283.323707] LustreError: 224645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679093 with bad export cookie 10372088357982397115 [ 3283.337255] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3289.602329] Lustre: server umount lustre-MDT0001 complete [ 3291.898533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3293.426301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3295.125548] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3296.573594] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3298.937145] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 18:25:07 (1756679107) [ 3301.327392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3301.467398] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3301.487152] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3301.595577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3301.597704] Lustre: Skipped 5 previous similar messages [ 3303.093630] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3306.242689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3306.279690] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3306.289881] Lustre: 231098:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3307.940605] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3310.596898] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3311.992089] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3314.786341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3314.869070] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3317.262837] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3319.631370] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3324.291441] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3324.338494] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3327.079393] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3329.576350] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3330.529427] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3332.320789] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3332.462849] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3334.219662] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3334.344753] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3335.699421] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff914e8874c800.ost_server_uuid 50 [ 3336.486113] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff914e8874c800.ost_server_uuid in FULL state after 0 sec [ 3336.923472] Lustre: Permanently deactivating lustre-OST0001 [ 3336.929870] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3338.068533] Lustre: Permanently reactivating lustre-OST0001 [ 3341.289054] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3341.289764] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3341.293957] LustreError: Skipped 4 previous similar messages [ 3341.301099] Lustre: Skipped 6 previous similar messages [ 3345.545356] Lustre: server umount lustre-OST0001 complete [ 3346.411281] LustreError: 232723:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3346.421460] LustreError: 232723:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 25 previous similar messages [ 3353.506504] Lustre: server umount lustre-OST0000 complete [ 3361.591077] Lustre: server umount lustre-MDT0000 complete [ 3363.206213] LustreError: 231088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679172 with bad export cookie 10372088357982398375 [ 3363.209688] LustreError: 231088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3363.211087] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3363.372980] Lustre: server umount lustre-MDT0001 complete [ 3365.277100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3366.607426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3367.886357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3369.207299] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3371.403762] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 18:26:20 (1756679180) [ 3371.792655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3374.235332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3374.345401] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3374.355502] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3374.358426] Lustre: Skipped 5 previous similar messages [ 3375.986121] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3379.533393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3379.567778] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3379.577753] Lustre: 237248:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3381.182983] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3382.966411] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3384.383502] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3387.368502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3387.416546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3387.540829] Lustre: Permanently deactivating lustre-OST0000 [ 3387.549956] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3388.601753] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3388.607426] LustreError: 239044:0:(osp_precreate.c:969:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -5 [ 3389.640713] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3391.374723] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3393.900348] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3396.154752] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3396.530400] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3398.080865] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3398.648137] Lustre: Permanently reactivating lustre-OST0000 [ 3401.912290] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3401.914741] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3401.921334] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3401.942935] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3401.950664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3417.147159] Lustre: server umount lustre-OST0001 complete [ 3424.946071] Lustre: server umount lustre-OST0000 complete [ 3432.873585] Lustre: server umount lustre-MDT0000 complete [ 3434.382786] LustreError: 238894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679244 with bad export cookie 10372088357982400013 [ 3434.387410] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3434.540267] Lustre: server umount lustre-MDT0001 complete [ 3438.958386] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 3440.277236] Key type lgssc unregistered [ 3440.444584] LNet: 241660:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3440.450441] LNetError: 241660:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3440.462751] LNet: Removed LNI 192.168.201.118@tcp [ 3440.936114] Key type .llcrypt unregistered [ 3440.937836] Key type ._llcrypt unregistered [ 3450.300192] Key type ._llcrypt registered [ 3450.301202] Key type .llcrypt registered [ 3450.347678] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3450.777887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3450.833828] alg: No test for adler32 (adler32-zlib) [ 3451.714794] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 3451.826097] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 3453.423174] Key type lgssc registered [ 3453.892558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3455.233968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3457.378634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3457.384769] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3458.582815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3459.978475] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3462.916932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3462.939159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3462.970181] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3462.979326] Lustre: 243074:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3462.982797] Lustre: Permanently deactivating lustre-MDT0001 [ 3462.987569] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3463.097200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3464.491276] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3466.514785] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3467.944319] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3468.283381] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3470.595497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3470.717995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3472.742677] LustreError: 244706:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3472.748377] LustreError: 244706:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 3472.750413] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3472.756249] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3473.763275] LustreError: 245047:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3474.726577] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3476.890526] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3476.965148] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3478.872733] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3481.048676] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3482.090587] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3485.168907] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3485.169519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3487.072854] Lustre: Permanently reactivating lustre-MDT0001 [ 3487.082803] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3487.087258] Lustre: Skipped 2 previous similar messages [ 3488.083488] Lustre: lustre-MDT0001: Client ee4db999-eb19-4acb-a6a2-1fa2b263ef81 (at 192.168.201.18@tcp) reconnecting [ 3489.373922] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3489.916272] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3491.615944] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3491.695918] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3491.842862] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3491.847572] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 3491.851608] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 3491.855602] LustreError: 246661:0:(llog_osd.c:2254:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 3491.856300] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3491.858057] LustreError: 246661:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 3491.862861] LustreError: 246661:0:(obd_config.c:1996:class_config_llog_handler()) MGC192.168.201.118@tcp: cfg command failed: rc = -108 [ 3491.865995] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 3491.865995] [ 3491.868440] LustreError: 243077:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 3493.092871] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3493.193498] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3494.350003] Lustre: Permanently deactivating lustre-MDT0001 [ 3494.361550] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3494.366704] Lustre: Skipped 2 previous similar messages [ 3497.403203] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3504.096798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3504.102245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3504.111925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3506.660540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3506.661287] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3506.665691] Lustre: Skipped 1 previous similar message [ 3509.311164] LustreError: 247200:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3509.345072] Lustre: server umount lustre-MDT0000 complete [ 3510.740066] LustreError: 243067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679320 with bad export cookie 7744457586429559007 [ 3510.744848] LustreError: 243067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3510.746297] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3510.822494] LustreError: 247401:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3510.824270] LustreError: 247401:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3510.907821] Lustre: server umount lustre-MDT0001 complete [ 3518.188297] LustreError: 247601:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3518.192928] LustreError: 247601:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3518.217719] Lustre: server umount lustre-OST0000 complete [ 3522.583546] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 18:28:51 (1756679331) [ 3526.728915] LustreError: 248607:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3526.730619] LustreError: 248607:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3526.802219] Lustre: server umount lustre-OST0001 complete [ 3529.182039] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 3532.397039] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3536.784631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3539.117052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3541.278681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3543.404858] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3546.045600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3546.071107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3546.176930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3546.181240] Lustre: Skipped 2 previous similar messages [ 3546.191906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3546.233059] Lustre: lustre-MDT0000: new disk, initializing [ 3546.260600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3546.266962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3547.531447] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3552.389517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3552.414193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3552.445490] Lustre: 250789:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3552.478354] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3552.479189] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3552.480211] Lustre: Skipped 3 previous similar messages [ 3552.483593] Lustre: Skipped 1 previous similar message [ 3552.519306] Lustre: lustre-MDT0001: new disk, initializing [ 3552.561888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3552.566497] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3553.791337] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3557.076122] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3558.268322] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3561.559727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3561.608683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3561.774358] Lustre: lustre-OST0000: new disk, initializing [ 3561.778241] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3561.835137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3561.839148] Lustre: Skipped 1 previous similar message [ 3564.273630] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3564.366871] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3564.371372] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3564.427207] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3567.975322] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3568.775134] LustreError: 253142:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3573.506640] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3573.540921] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3573.610932] Lustre: lustre-OST0001: new disk, initializing [ 3573.613703] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3576.204453] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3578.871087] LustreError: 253142:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3578.879071] LustreError: 253142:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3579.496244] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3580.054582] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3582.461720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3582.467114] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3582.499056] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3588.967206] LustreError: 253142:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3592.681697] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3592.682756] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3592.685920] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3592.687889] Lustre: Skipped 2 previous similar messages [ 3592.696096] Lustre: Skipped 1 previous similar message [ 3595.826160] LustreError: 254286:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3595.828860] LustreError: 254286:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3595.851926] Lustre: server umount lustre-OST0001 complete [ 3597.796580] LustreError: 252520:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3597.800899] LustreError: 252520:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 3602.915555] LustreError: 252519:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3602.919899] LustreError: 252519:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 3608.035682] LustreError: 252892:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3608.042542] LustreError: 252892:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 3610.592445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3610.592678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3610.595579] LustreError: Skipped 2 previous similar messages [ 3610.595994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3610.598670] Lustre: Skipped 1 previous similar message [ 3618.277538] LustreError: 252518:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3618.277539] LustreError: 252519:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3618.281512] LustreError: 252518:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 3618.286991] LustreError: 252519:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 3622.367096] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3622.386555] LustreError: 254507:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3622.388334] LustreError: 254507:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3622.452375] Lustre: server umount lustre-OST0000 complete [ 3624.416032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3624.418375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3624.422314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3624.425529] Lustre: Skipped 5 previous similar messages [ 3638.239122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3638.304885] Lustre: server umount lustre-MDT0000 complete [ 3639.581203] LustreError: 251570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679449 with bad export cookie 7744457586429561828 [ 3639.583978] LustreError: 251570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3639.584541] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3645.711603] LustreError: 254914:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3645.714954] LustreError: 254914:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3645.847014] Lustre: server umount lustre-MDT0001 complete [ 3650.453614] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 3651.885415] Key type lgssc unregistered [ 3652.076613] LNet: 255445:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3652.081402] LNetError: 255445:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3652.090766] LNet: Removed LNI 192.168.201.118@tcp [ 3652.588437] Key type .llcrypt unregistered [ 3652.589663] Key type ._llcrypt unregistered [ 3666.238619] Key type ._llcrypt registered [ 3666.240284] Key type .llcrypt registered [ 3666.300495] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 3672.571560] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3673.076923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3673.088375] alg: No test for adler32 (adler32-zlib) [ 3673.994377] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 3674.096358] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 3675.687242] Key type lgssc registered [ 3676.160219] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3679.932265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3682.374647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3684.893535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3687.199528] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3689.901601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3689.933792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3689.943735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3691.078868] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3691.097514] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3691.137465] Lustre: lustre-MDT0000: new disk, initializing [ 3691.169868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3691.177533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3692.723483] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3697.764497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3697.783285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3697.811741] Lustre: 259369:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3697.832610] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3697.834144] Lustre: Skipped 1 previous similar message [ 3697.870433] Lustre: lustre-MDT0001: new disk, initializing [ 3697.901974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3697.913780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3697.916537] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3699.450061] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3702.920665] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3704.026700] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3706.519218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3706.545877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3706.655597] Lustre: lustre-OST0000: new disk, initializing [ 3706.660116] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3706.700043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3707.284413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3707.290295] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3707.339095] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3708.860332] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3712.280305] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3713.611457] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3713.686958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3715.119674] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3715.187066] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3717.601105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3717.602052] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3717.603659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3717.603809] LustreError: Skipped 1 previous similar message [ 3717.607592] Lustre: Skipped 1 previous similar message [ 3717.997237] LustreError: 262306:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3718.067715] Lustre: server umount lustre-OST0000 complete [ 3722.722362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3722.731768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3722.734762] Lustre: Skipped 2 previous similar messages [ 3725.759448] LustreError: 262507:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3725.761779] LustreError: 262507:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3725.794346] Lustre: server umount lustre-MDT0000 complete [ 3727.074564] LustreError: 260150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679536 with bad export cookie 13723304922103324391 [ 3727.078577] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3727.079024] LustreError: 260150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3727.147531] LustreError: 262710:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3727.150183] LustreError: 262710:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3727.235245] Lustre: server umount lustre-MDT0001 complete [ 3730.135975] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 18:32:18 (1756679538) [ 3732.520154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3732.706470] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3734.076505] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3737.284632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3737.491972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3738.877178] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3741.268891] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3742.617594] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3745.256200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3747.548971] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3749.380619] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3799.016672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3799.017129] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3799.019685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3799.021306] Lustre: Skipped 2 previous similar messages [ 3799.023391] LustreError: Skipped 1 previous similar message [ 3804.133105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3804.137053] Lustre: Skipped 1 previous similar message [ 3804.272133] LustreError: 265721:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3804.277317] LustreError: 265721:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3804.395534] Lustre: server umount lustre-OST0000 complete [ 3806.460307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3811.167805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3811.304545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3811.306285] Lustre: Skipped 1 previous similar message [ 3813.423345] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3815.430773] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3828.711331] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3828.714757] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3828.714889] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3828.719150] Lustre: Skipped 1 previous similar message [ 3854.307171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3854.310905] LustreError: Skipped 1 previous similar message [ 3854.310951] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3854.313591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3854.319887] Lustre: Skipped 1 previous similar message [ 3855.404145] LustreError: 267386:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3855.405821] LustreError: 267386:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3855.491360] Lustre: server umount lustre-OST0000 complete [ 3859.424706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3863.152440] Lustre: server umount lustre-MDT0000 complete [ 3864.466138] LustreError: 265621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679674 with bad export cookie 13723304922103325273 [ 3864.469052] LustreError: 265621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3864.469345] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3864.534376] LustreError: 267789:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3864.537105] LustreError: 267789:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3864.632623] Lustre: server umount lustre-MDT0001 complete [ 3868.854707] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 3869.860125] Key type lgssc unregistered [ 3870.015613] LNet: 268320:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3870.018341] LNetError: 268320:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3870.025628] LNet: Removed LNI 192.168.201.118@tcp [ 3870.463478] Key type .llcrypt unregistered [ 3870.465448] Key type ._llcrypt unregistered [ 3880.702344] Key type ._llcrypt registered [ 3880.704268] Key type .llcrypt registered [ 3880.747534] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3881.145330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3881.213779] alg: No test for adler32 (adler32-zlib) [ 3882.136862] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 3882.240305] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 3883.855342] Key type lgssc registered [ 3884.527648] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3887.838107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3887.851341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3889.088956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3890.469582] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3893.873898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3894.043572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3895.393838] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3897.464298] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3898.699927] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3901.306869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3901.427881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3903.539439] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3905.634255] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3913.710324] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3929.060901] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3929.061686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3929.066406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3929.074434] Lustre: Skipped 1 previous similar message [ 3930.797514] LustreError: 272734:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3930.889149] Lustre: server umount lustre-OST0000 complete [ 3934.177318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3934.178107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3934.181389] Lustre: Skipped 1 previous similar message [ 3934.184749] Lustre: Skipped 1 previous similar message [ 3938.379531] LustreError: 272937:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3938.383195] LustreError: 272937:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3938.418674] Lustre: server umount lustre-MDT0000 complete [ 3939.296838] LustreError: 270971:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3939.306897] LustreError: 270971:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 3940.116689] LustreError: 269676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679749 with bad export cookie 9139007264724293240 [ 3940.122511] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3940.123047] LustreError: 269676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3940.198610] LustreError: 273139:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3940.201715] LustreError: 273139:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3940.293036] Lustre: server umount lustre-MDT0001 complete [ 3944.143755] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 3945.225820] Key type lgssc unregistered [ 3945.352333] LNet: 273671:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3945.354537] LNetError: 273671:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3945.365299] LNet: Removed LNI 192.168.201.118@tcp [ 3945.653359] Key type .llcrypt unregistered [ 3945.654513] Key type ._llcrypt unregistered [ 3952.531489] Key type ._llcrypt registered [ 3952.532460] Key type .llcrypt registered [ 3952.564685] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 3952.865552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3952.892072] alg: No test for adler32 (adler32-zlib) [ 3953.734044] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 3953.812300] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 3955.391179] Key type lgssc registered [ 3955.748948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3958.406397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3958.414284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3959.601667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3960.975418] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3963.860428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3963.993127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3965.460069] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3967.201678] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3968.396324] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3970.853865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3970.964225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3973.149125] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 3975.152806] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3984.300802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3994.593662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3994.595302] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3994.599397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3994.602978] Lustre: Skipped 1 previous similar message [ 3995.629597] LustreError: 277263:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3995.694137] Lustre: server umount lustre-OST0000 complete [ 3999.713981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3999.714123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3999.720222] Lustre: Skipped 1 previous similar message [ 3999.721230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3999.724739] LustreError: Skipped 1 previous similar message [ 3999.732444] Lustre: Skipped 1 previous similar message [ 4003.194990] LustreError: 277465:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4003.197032] LustreError: 277465:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4003.225931] Lustre: server umount lustre-MDT0000 complete [ 4004.606615] LustreError: 276470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679814 with bad export cookie 16645633042325611470 [ 4004.609290] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4004.610477] LustreError: 276470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4004.684078] LustreError: 277666:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4004.685858] LustreError: 277666:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4004.769641] Lustre: server umount lustre-MDT0001 complete [ 4009.322345] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 4010.671057] Key type lgssc unregistered [ 4010.798320] LNet: 278197:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4010.800665] LNetError: 278197:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4010.807286] LNet: Removed LNI 192.168.201.118@tcp [ 4011.072133] Key type .llcrypt unregistered [ 4011.073296] Key type ._llcrypt unregistered [ 4020.442588] Key type ._llcrypt registered [ 4020.443495] Key type .llcrypt registered [ 4020.482432] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4021.012396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4021.048232] alg: No test for adler32 (adler32-zlib) [ 4021.920662] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 4022.009393] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 4023.591233] Key type lgssc registered [ 4024.019086] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4027.427202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4027.440364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4028.640253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4030.237615] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4033.745102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4033.863881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4035.282336] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4037.180253] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4038.651801] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4041.340102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4041.503485] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4043.957736] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4046.389048] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4053.808510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4069.349009] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4069.351568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4069.357330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4069.359995] LustreError: Skipped 1 previous similar message [ 4072.688197] LustreError: 282607:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4072.785762] Lustre: server umount lustre-OST0000 complete [ 4074.464319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4074.464913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4074.467405] Lustre: Skipped 2 previous similar messages [ 4074.469919] Lustre: Skipped 2 previous similar messages [ 4079.584300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4079.586014] Lustre: Skipped 1 previous similar message [ 4080.458739] LustreError: 282810:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4080.460962] LustreError: 282810:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4080.489792] Lustre: server umount lustre-MDT0000 complete [ 4081.836857] LustreError: 279551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679891 with bad export cookie 8276169306239697690 [ 4081.842811] LustreError: 279551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4081.844195] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4081.927554] LustreError: 283012:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4081.930196] LustreError: 283012:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4082.029506] Lustre: server umount lustre-MDT0001 complete [ 4086.462594] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 4087.953718] Key type lgssc unregistered [ 4088.093948] LNet: 283543:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4088.096387] LNetError: 283543:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4088.105627] LNet: Removed LNI 192.168.201.118@tcp [ 4088.494659] Key type .llcrypt unregistered [ 4088.496765] Key type ._llcrypt unregistered [ 4096.263685] Key type ._llcrypt registered [ 4096.265679] Key type .llcrypt registered [ 4096.315870] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4096.874647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4096.939655] alg: No test for adler32 (adler32-zlib) [ 4097.870739] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 4097.972594] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 4099.567219] Key type lgssc registered [ 4100.136708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4102.935789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4102.944449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4104.157372] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4105.482725] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4108.857349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4109.024937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4110.522402] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4112.692940] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4114.121317] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4116.762887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4116.927008] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4119.349907] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4121.170443] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4129.192288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4139.494143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4139.494859] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4139.502780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4139.504418] LustreError: Skipped 1 previous similar message [ 4139.507334] Lustre: Skipped 1 previous similar message [ 4141.873296] LustreError: 287173:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4141.955812] Lustre: server umount lustre-OST0000 complete [ 4144.611727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4144.619141] Lustre: Skipped 1 previous similar message [ 4144.622716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4144.626661] Lustre: Skipped 1 previous similar message [ 4149.455046] LustreError: 287382:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4149.458079] LustreError: 287382:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4149.489536] Lustre: server umount lustre-MDT0000 complete [ 4149.728636] LustreError: 284750:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4149.735820] LustreError: 284750:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 4150.973207] LustreError: 287175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756679960 with bad export cookie 16677260836113479077 [ 4150.976961] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4150.977861] LustreError: 287175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4151.049876] LustreError: 287584:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4151.051563] LustreError: 287584:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4151.138993] Lustre: server umount lustre-MDT0001 complete [ 4154.803708] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 4155.959841] Key type lgssc unregistered [ 4156.121484] LNet: 288115:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4156.123818] LNetError: 288115:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4156.134419] LNet: Removed LNI 192.168.201.118@tcp [ 4156.455325] Key type .llcrypt unregistered [ 4156.456293] Key type ._llcrypt unregistered [ 4160.712738] Key type ._llcrypt registered [ 4160.714283] Key type .llcrypt registered [ 4160.750619] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4167.440947] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 4174.121661] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4174.601691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4174.613506] alg: No test for adler32 (adler32-zlib) [ 4175.472348] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 4175.558632] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 4177.143195] Key type lgssc registered [ 4177.488388] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4180.571296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4183.011260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4185.711763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4188.093185] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4190.837774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4190.857516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4190.862059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4191.959290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4191.973738] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4192.016947] Lustre: lustre-MDT0000: new disk, initializing [ 4192.046755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4192.054802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4193.475146] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4198.439427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4198.481538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4198.552866] Lustre: 291601:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4198.589848] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4198.592738] Lustre: Skipped 1 previous similar message [ 4198.642389] Lustre: lustre-MDT0001: new disk, initializing [ 4198.699701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4198.724499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4198.730488] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4200.425277] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4204.203971] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4205.838340] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4208.462305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4208.487426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4208.628089] Lustre: lustre-OST0000: new disk, initializing [ 4208.630663] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4208.665165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4210.697913] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4212.549088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4212.552195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4212.595624] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4214.070258] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4215.701459] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4215.783735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4217.391560] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4217.457834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4222.945403] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4222.945943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4222.947243] LustreError: Skipped 1 previous similar message [ 4222.948072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4222.952709] Lustre: Skipped 1 previous similar message [ 4224.110083] LustreError: 294540:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4224.180374] Lustre: server umount lustre-OST0000 complete [ 4228.064749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4228.065692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4228.070721] Lustre: Skipped 1 previous similar message [ 4228.072738] Lustre: Skipped 1 previous similar message [ 4231.618342] LustreError: 294741:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4231.620835] LustreError: 294741:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4231.647805] Lustre: server umount lustre-MDT0000 complete [ 4232.959714] LustreError: 293348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680042 with bad export cookie 8017283971416916209 [ 4232.962738] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4232.967114] LustreError: 293348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4233.040075] LustreError: 294943:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4233.043985] LustreError: 294943:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4233.144241] Lustre: server umount lustre-MDT0001 complete [ 4236.361161] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 18:40:45 (1756680045) [ 4238.968277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4239.215629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4240.759897] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4243.870044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4244.002711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4245.502668] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4247.478169] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4248.676210] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4251.309540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4253.963270] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4256.047032] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4270.053802] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4270.056210] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4270.060112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4270.067643] Lustre: Skipped 1 previous similar message [ 4270.074218] Lustre: Skipped 2 previous similar messages [ 4275.176182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4275.179786] Lustre: Skipped 1 previous similar message [ 4275.757064] LustreError: 297883:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4275.758953] LustreError: 297883:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4275.833573] Lustre: server umount lustre-OST0000 complete [ 4279.776070] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4279.777415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4279.778075] LustreError: Skipped 1 previous similar message [ 4279.786291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4283.276422] LustreError: 298088:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4283.279099] LustreError: 298088:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4283.314210] Lustre: server umount lustre-MDT0000 complete [ 4284.555043] LustreError: 295592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680094 with bad export cookie 8017283971416917091 [ 4284.557531] LustreError: 296335:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4284.558257] LustreError: 295592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4284.558711] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4284.698847] Lustre: server umount lustre-MDT0001 complete [ 4288.727776] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 4289.902543] Key type lgssc unregistered [ 4290.056650] LNet: 298822:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4290.060484] LNetError: 298822:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4290.073642] LNet: Removed LNI 192.168.201.118@tcp [ 4290.445380] Key type .llcrypt unregistered [ 4290.446462] Key type ._llcrypt unregistered [ 4299.279167] Key type ._llcrypt registered [ 4299.280422] Key type .llcrypt registered [ 4299.320970] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 4305.196903] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4305.509602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4305.546101] alg: No test for adler32 (adler32-zlib) [ 4306.407543] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 4306.499383] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 4308.079219] Key type lgssc registered [ 4308.498386] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4311.419496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4314.047978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4316.264455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4318.960818] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4320.847886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4322.747276] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4325.204325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4325.238595] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4325.248623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4326.341644] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4326.353386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4326.394150] Lustre: lustre-MDT0000: new disk, initializing [ 4326.420893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4326.428147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4327.977847] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4332.612031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4332.645435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4332.679580] Lustre: 302598:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4332.708745] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4332.711548] Lustre: Skipped 1 previous similar message [ 4332.752045] Lustre: lustre-MDT0001: new disk, initializing [ 4332.794523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4332.806829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4332.811387] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4334.381377] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4337.994311] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4339.500078] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4342.009602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4342.043176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4342.188984] Lustre: lustre-OST2710: new disk, initializing [ 4342.191871] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4342.240406] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4343.887880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4343.894723] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4343.961531] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4344.928266] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4348.744661] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4351.837761] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4351.884830] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4351.993927] Lustre: lustre-OST03e8: new disk, initializing [ 4351.996922] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4352.046754] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4353.130985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4353.136281] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4353.197563] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4354.292184] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4356.231470] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4359.257647] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4359.374253] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4361.017637] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4361.103966] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4362.571740] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4362.694161] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4364.634851] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4364.761668] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4368.868347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4368.868930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4368.871175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4368.871913] Lustre: Skipped 1 previous similar message [ 4372.054558] LustreError: 306940:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4372.096458] Lustre: server umount lustre-MDT0000 complete [ 4373.872946] LustreError: 306942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680183 with bad export cookie 1368607571187859998 [ 4373.880574] LustreError: 306942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4373.882697] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4373.973604] LustreError: 307143:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4373.977174] LustreError: 307143:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4373.988739] Lustre: lustre-MDT0001-lwp-OST2710: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4373.992397] Lustre: Skipped 2 previous similar messages [ 4374.094174] Lustre: server umount lustre-MDT0001 complete [ 4381.933811] LustreError: 307343:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4381.936553] LustreError: 307343:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4381.953222] Lustre: server umount lustre-OST2710 complete [ 4383.599487] Lustre: server umount lustre-OST03e8 complete [ 4390.222344] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 4393.749994] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4398.179163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4401.050315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4403.749316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4406.537529] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4409.371641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4409.417162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4409.550342] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4409.568699] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4409.621091] Lustre: lustre-MDT0000: new disk, initializing [ 4409.651240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4409.660429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4411.103965] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4416.498520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4416.546094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4416.604885] Lustre: 310337:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4416.770351] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4418.311169] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4422.261768] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4423.935594] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4426.501213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4426.542910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4426.671081] Lustre: lustre-OST0000: new disk, initializing [ 4426.674530] Lustre: Skipped 1 previous similar message [ 4426.678483] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4426.682589] Lustre: Skipped 2 previous similar messages [ 4426.732327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4426.736230] Lustre: Skipped 1 previous similar message [ 4429.312343] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4433.010561] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4433.923644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4433.932575] Lustre: Skipped 1 previous similar message [ 4433.937346] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4433.987226] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4434.717538] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4434.857176] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4436.789688] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4436.914352] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4439.010819] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4439.012203] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4439.014859] LustreError: Skipped 1 previous similar message [ 4439.021931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4439.027829] Lustre: Skipped 4 previous similar messages [ 4443.951305] LustreError: 313274:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4443.954762] LustreError: 313274:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4444.045604] Lustre: server umount lustre-OST0000 complete [ 4447.712429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4447.716204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4447.721561] Lustre: Skipped 1 previous similar message [ 4447.724064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4451.840681] LustreError: 313476:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4451.844428] LustreError: 313476:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4451.879391] Lustre: server umount lustre-MDT0000 complete [ 4453.306830] LustreError: 310329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680263 with bad export cookie 1368607571187861664 [ 4453.313250] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4453.315756] LustreError: 310329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4453.470996] Lustre: server umount lustre-MDT0001 complete [ 4456.624809] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 18:44:25 (1756680265) [ 4457.278505] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4457.976901] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 18:44:26 (1756680266) [ 4459.126710] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4462.103621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4463.701918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4465.035701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4466.558965] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4466.979268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4469.410772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4469.544690] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4469.558337] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4469.624356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4470.992797] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4474.396468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4474.432708] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4474.442407] Lustre: 315803:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4476.055292] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4477.974488] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4479.283108] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4481.952558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4481.983162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4482.055977] Lustre: Denying initial registration attempt from nid 192.168.201.118@tcp, specified as failover [ 4482.060890] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4482.065804] LustreError: 317419:0:(tgt_mount.c:2460:server_fill_super()) Unable to start targets: -99 [ 4482.068996] LustreError: 317419:0:(tgt_mount.c:1954:server_put_super()) no obd lustre-OST0000 [ 4482.072407] LustreError: 317419:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4482.122338] Lustre: server umount lustre-OST0000 complete [ 4482.123746] LustreError: 317419:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4485.087822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4485.090025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4485.093294] Lustre: Skipped 1 previous similar message [ 4485.094850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4485.097792] Lustre: Skipped 2 previous similar messages [ 4490.695115] LustreError: 317786:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4490.699616] LustreError: 317786:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4490.734658] Lustre: server umount lustre-MDT0000 complete [ 4491.233586] LustreError: 315814:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4491.239168] LustreError: 315814:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 6 previous similar messages [ 4492.207419] LustreError: 315797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680301 with bad export cookie 1368607571187862546 [ 4492.211203] LustreError: 315797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4492.212684] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4492.371334] Lustre: server umount lustre-MDT0001 complete [ 4496.047280] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 4497.325128] Key type lgssc unregistered [ 4497.457439] LNet: 318520:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4497.459660] LNetError: 318520:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4497.469379] LNet: Removed LNI 192.168.201.118@tcp [ 4497.774090] Key type .llcrypt unregistered [ 4497.775075] Key type ._llcrypt unregistered [ 4501.366088] Key type ._llcrypt registered [ 4501.367481] Key type .llcrypt registered [ 4501.410965] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4501.854622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4501.871503] alg: No test for adler32 (adler32-zlib) [ 4502.744542] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 4502.842186] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 4504.431265] Key type lgssc registered [ 4504.962227] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4507.629747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4508.896060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4510.233499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4511.666146] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4512.107858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4518.781374] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4522.730805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4522.742462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4523.906733] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4523.922317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4524.019640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4525.448440] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4529.123255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4529.164944] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4529.181645] Lustre: 321253:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4529.319587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4530.684032] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4532.730544] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4534.365857] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4536.669331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4536.699362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4536.770597] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4536.841330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4539.377265] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4541.267477] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4543.971221] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4543.972157] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4543.973358] LustreError: Skipped 1 previous similar message [ 4543.974302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4543.976428] Lustre: Skipped 1 previous similar message [ 4543.981528] Lustre: Skipped 1 previous similar message [ 4548.012080] LustreError: 323533:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4548.081867] Lustre: server umount lustre-OST0000 complete [ 4550.111931] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4550.114041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4550.117731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4554.209528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4554.210406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4555.454091] LustreError: 323736:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4555.456954] LustreError: 323736:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4555.487869] Lustre: server umount lustre-MDT0000 complete [ 4556.772193] LustreError: 321244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680366 with bad export cookie 11697914797288422868 [ 4556.775664] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4556.776110] LustreError: 321244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4556.843759] LustreError: 323939:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4556.845627] LustreError: 323939:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4556.934789] Lustre: server umount lustre-MDT0001 complete [ 4560.585075] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 4561.756946] Key type lgssc unregistered [ 4561.896538] LNet: 324470:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4561.898777] LNetError: 324470:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4561.909404] LNet: Removed LNI 192.168.201.118@tcp [ 4562.228047] Key type .llcrypt unregistered [ 4562.229078] Key type ._llcrypt unregistered [ 4572.803910] Key type ._llcrypt registered [ 4572.804912] Key type .llcrypt registered [ 4572.851510] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4573.247523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4573.255852] alg: No test for adler32 (adler32-zlib) [ 4574.106116] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 4574.180488] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 4575.759147] Key type lgssc registered [ 4576.091605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4579.131603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4579.139777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4580.323688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4581.919290] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4585.670788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4585.861831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4587.583415] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4589.813593] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4591.465541] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4594.529076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4594.656623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4597.007376] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4598.982509] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4611.552201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4611.555502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4611.561525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4614.541634] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.18@tcp (stopping) [ 4616.162081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4616.162718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4616.165286] Lustre: Skipped 1 previous similar message [ 4616.171567] Lustre: Skipped 1 previous similar message [ 4617.469652] LustreError: 328113:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4617.504652] Lustre: server umount lustre-MDT0000 complete [ 4618.742153] LustreError: 325826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680428 with bad export cookie 12265025464520203028 [ 4618.744687] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4618.745748] LustreError: 325826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4619.655763] LustreError: 327053:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4619.660460] LustreError: 327053:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 4619.661779] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.18@tcp (stopping) [ 4619.666222] Lustre: Skipped 1 previous similar message [ 4621.280751] LustreError: 327053:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4621.280977] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4623.328186] LustreError: 325840:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4624.775578] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.18@tcp (stopping) [ 4624.778543] Lustre: Skipped 2 previous similar messages [ 4628.447835] LustreError: 325840:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4628.451807] LustreError: 325840:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 4633.055082] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4633.083741] LustreError: 328316:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4633.086310] LustreError: 328316:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4633.167084] Lustre: server umount lustre-MDT0001 complete [ 4635.519456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4638.455449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4643.807715] LustreError: 325154:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e45e6de8 x1842012889309312/t0(0) o250->MGC192.168.201.118@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4643.942089] LustreError: 327454:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4643.951449] LustreError: 327454:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 4643.968613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4643.991393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 4645.249930] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4648.365646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4648.494110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4649.927398] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4651.785899] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4658.663928] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4661.528106] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec [ 4662.603876] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4663.219556] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4664.800121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4664.804579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4664.810833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4664.812745] Lustre: Skipped 2 previous similar messages [ 4668.900606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4668.902845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4670.776237] LustreError: 330620:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4670.779879] LustreError: 330620:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4670.860448] Lustre: server umount lustre-OST0000 complete [ 4674.017649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4678.205924] LustreError: 330821:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4678.208263] LustreError: 330821:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4678.237094] Lustre: server umount lustre-MDT0000 complete [ 4679.136034] LustreError: 330822:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4679.142805] LustreError: 330822:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 4679.521408] LustreError: 327465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680489 with bad export cookie 12265025464520210147 [ 4679.526133] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4679.527229] LustreError: 327465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4679.673027] Lustre: server umount lustre-MDT0001 complete [ 4683.072725] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 4684.287733] Key type lgssc unregistered [ 4684.431456] LNet: 331553:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4684.434130] LNetError: 331553:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4684.441617] LNet: Removed LNI 192.168.201.118@tcp [ 4684.853183] Key type .llcrypt unregistered [ 4684.854889] Key type ._llcrypt unregistered [ 4694.431382] Key type ._llcrypt registered [ 4694.433086] Key type .llcrypt registered [ 4694.484686] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4694.876266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4694.883909] alg: No test for adler32 (adler32-zlib) [ 4695.756791] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 4695.857524] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 4697.455157] Key type lgssc registered [ 4697.860132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4700.957277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4700.964364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4702.135371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4703.488803] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4706.363912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4706.501330] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4707.869275] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4709.655948] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4710.823419] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4711.903947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4711.906505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4711.910525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4717.024773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4717.026067] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4722.145222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4722.148419] Lustre: Skipped 2 previous similar messages [ 4726.239255] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4726.293411] LustreError: 334316:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4726.338721] Lustre: server umount lustre-MDT0000 complete [ 4727.264926] LustreError: 332929:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4727.275403] LustreError: 332929:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 4728.113443] LustreError: 332909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680537 with bad export cookie 14315301882854828307 [ 4728.119764] LustreError: 332909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4728.122180] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4728.218634] LustreError: 334522:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4728.221582] LustreError: 334522:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4728.342691] Lustre: server umount lustre-MDT0001 complete [ 4734.559040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4734.655225] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4734.666796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4734.742616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4736.135232] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4739.748449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4739.784873] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4739.796455] Lustre: 335291:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4739.894155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4741.235123] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4743.542597] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4745.273244] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4748.357625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4748.459818] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4748.464900] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4748.469416] LustreError: 336950:0:(tgt_mount.c:2460:server_fill_super()) Unable to start targets: -2 [ 4748.472976] LustreError: 336950:0:(tgt_mount.c:1954:server_put_super()) no obd lustre-OST0000 [ 4748.475869] LustreError: 336950:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4748.542294] Lustre: server umount lustre-OST0000 complete [ 4748.544121] LustreError: 336950:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4751.549864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4751.634541] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4751.707740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4753.824409] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4755.760733] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4758.579478] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4758.602795] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4758.624280] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4758.648422] Lustre: lustre-OST0001: new disk, initializing [ 4758.650141] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4759.629844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4759.632430] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4759.658025] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 4759.681837] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4760.648138] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4762.468697] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4765.158942] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4765.159539] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4765.162520] LustreError: Skipped 1 previous similar message [ 4765.168731] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4765.173342] Lustre: Skipped 1 previous similar message [ 4765.485226] LustreError: 339063:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4765.486953] LustreError: 339063:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4765.505292] Lustre: server umount lustre-OST0001 complete [ 4769.759932] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4769.762048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4769.765351] Lustre: Skipped 1 previous similar message [ 4769.768223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4770.275107] LustreError: 337415:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4770.284076] LustreError: 337415:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 4772.975298] LustreError: 339264:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4772.977902] LustreError: 339264:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4773.064798] Lustre: server umount lustre-OST0000 complete [ 4775.393554] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4775.397518] Lustre: Skipped 2 previous similar messages [ 4780.512982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4780.515980] Lustre: Skipped 5 previous similar messages [ 4780.554767] LustreError: 339466:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4780.557974] LustreError: 339466:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4780.594811] Lustre: server umount lustre-MDT0000 complete [ 4781.868508] LustreError: 335282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680591 with bad export cookie 14315301882854828846 [ 4781.873479] LustreError: 335282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4781.875706] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4782.052772] Lustre: server umount lustre-MDT0001 complete [ 4786.060960] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 4787.490533] Key type lgssc unregistered [ 4787.687828] LNet: 340198:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4787.692909] LNetError: 340198:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4787.702822] LNet: Removed LNI 192.168.201.118@tcp [ 4788.196887] Key type .llcrypt unregistered [ 4788.198798] Key type ._llcrypt unregistered [ 4799.946076] Key type ._llcrypt registered [ 4799.947543] Key type .llcrypt registered [ 4799.989826] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 4806.434422] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4806.758668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4806.784953] alg: No test for adler32 (adler32-zlib) [ 4807.650053] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 4807.753357] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 4809.335147] Key type lgssc registered [ 4809.831904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4814.413357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4816.911854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4819.008195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4821.701183] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4824.059423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4824.078992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4824.084201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4825.177427] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4825.190586] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4825.231158] Lustre: lustre-MDT0000: new disk, initializing [ 4825.255790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4825.262551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4826.628322] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4830.641569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4830.664525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4830.700038] Lustre: 344122:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4830.723592] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4830.725367] Lustre: Skipped 1 previous similar message [ 4830.761244] Lustre: lustre-MDT0001: new disk, initializing [ 4830.795301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4830.808909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4830.812714] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4832.201584] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4835.913523] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4837.369252] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4839.712835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4839.744711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4839.873527] Lustre: lustre-OST0000: new disk, initializing [ 4839.877341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4839.913167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4842.272792] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4845.725956] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4847.336664] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4847.608970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4847.611818] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4847.644093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4848.431337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4849.787356] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4849.873018] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4852.710197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4852.713972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4852.717262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4852.719705] Lustre: Skipped 1 previous similar message [ 4856.751157] LustreError: 347067:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4856.863806] Lustre: server umount lustre-OST0000 complete [ 4861.408538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4861.413162] LustreError: Skipped 1 previous similar message [ 4861.416041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4861.424647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4861.428399] Lustre: Skipped 1 previous similar message [ 4862.944279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4862.944736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4864.590058] LustreError: 347269:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4864.594684] LustreError: 347269:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4864.632864] Lustre: server umount lustre-MDT0000 complete [ 4865.858431] LustreError: 344905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680675 with bad export cookie 7486314336015165814 [ 4865.862061] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4865.862670] LustreError: 344905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4865.917850] LustreError: 347471:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4865.920387] LustreError: 347471:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4866.021207] Lustre: server umount lustre-MDT0001 complete [ 4869.810683] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 18:51:18 (1756680678) [ 4872.282970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4875.199306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4885.046702] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 4889.204734] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 4892.988510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4895.118747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4897.031450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4898.977205] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4901.076108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4901.098762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4901.183900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4901.195471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4901.233807] Lustre: lustre-MDT0000: new disk, initializing [ 4901.256922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4901.263880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4902.451762] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4906.472979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4906.497376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4906.527629] Lustre: 351239:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4906.554647] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4906.556224] Lustre: Skipped 1 previous similar message [ 4906.594522] Lustre: lustre-MDT0001: new disk, initializing [ 4906.630931] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4906.642118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4906.644720] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4907.897447] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4910.885668] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4912.028208] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4914.301643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4914.329859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4915.839251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4915.842454] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4915.888144] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4916.265170] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4919.258314] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4920.641515] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4920.725561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4922.033906] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4922.126371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4926.432695] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4926.434745] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4926.434774] LustreError: Skipped 1 previous similar message [ 4926.438594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4926.440786] Lustre: Skipped 2 previous similar messages [ 4931.555498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4931.557363] Lustre: Skipped 1 previous similar message [ 4937.183150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4937.202101] LustreError: 354177:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4937.203932] LustreError: 354177:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4937.282334] Lustre: server umount lustre-OST0000 complete [ 4941.794229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4941.794976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4941.797506] Lustre: Skipped 2 previous similar messages [ 4941.800254] Lustre: Skipped 3 previous similar messages [ 4944.570326] LustreError: 354379:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4944.572173] LustreError: 354379:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4944.603050] Lustre: server umount lustre-MDT0000 complete [ 4946.096204] LustreError: 352019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680755 with bad export cookie 7486314336015166682 [ 4946.103013] LustreError: 352019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4946.104356] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4946.308159] Lustre: server umount lustre-MDT0001 complete [ 4950.051091] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 18:52:38 (1756680758) [ 4952.486328] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 18:52:41 (1756680761) [ 4955.664579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4955.881869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4955.885245] Lustre: Skipped 1 previous similar message [ 4957.411920] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4960.930795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4962.382967] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4964.592192] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4965.746386] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4968.071214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4970.224094] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 4971.937728] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4986.161830] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4986.765217] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4987.332819] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4987.976880] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4988.707248] Lustre: Failing over lustre-MDT0000 [ 4988.968975] LustreError: 358056:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4988.970868] LustreError: 358056:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4988.999473] Lustre: server umount lustre-MDT0000 complete [ 4990.946316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4990.949335] Lustre: Skipped 2 previous similar messages [ 4990.949337] LustreError: 355589:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4990.953925] LustreError: 355589:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 4 previous similar messages [ 4992.917643] LustreError: 357550:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.18@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4996.066793] LustreError: 355593:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4996.072255] LustreError: 355593:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 5001.186111] LustreError: 355589:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5001.192823] LustreError: 355589:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 5002.055519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5002.092218] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5002.185144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5002.186729] Lustre: Skipped 2 previous similar messages [ 5002.218408] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5003.144590] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5003.413719] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5007.336807] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5007.343601] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5007.360573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5008.814330] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5009.580225] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5011.489692] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5012.121261] Lustre: Failing over lustre-MDT0000 [ 5012.261284] LustreError: 359134:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5012.262829] LustreError: 359134:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5012.295141] Lustre: server umount lustre-MDT0000 complete [ 5012.452557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5012.453429] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5012.455185] LustreError: 355594:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5012.460071] Lustre: Skipped 2 previous similar messages [ 5012.468651] LustreError: 355594:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 5022.691296] LustreError: 355594:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5022.696991] LustreError: 355594:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 7 previous similar messages [ 5025.987394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5026.027186] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5026.149023] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5027.471160] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5028.769940] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5031.396723] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5031.399243] Lustre: Skipped 2 previous similar messages [ 5031.406396] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5031.423275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5032.884921] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5033.554866] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5036.516886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5036.517799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5036.523807] Lustre: Skipped 2 previous similar messages [ 5042.105904] Lustre: server umount lustre-MDT0000 complete [ 5043.797652] LustreError: 355620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680853 with bad export cookie 7486314336015169321 [ 5043.804628] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5043.991115] Lustre: server umount lustre-MDT0001 complete [ 5055.730187] LustreError: 360565:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5055.732292] LustreError: 360565:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5055.808208] Lustre: server umount lustre-OST0000 complete [ 5061.023408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5061.192117] LustreError: 361392:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5061.197323] LustreError: 361392:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 5061.214961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5061.217332] Lustre: Skipped 1 previous similar message [ 5062.484742] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5065.251444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5066.688993] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5068.472120] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5069.636047] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5072.044812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5074.191692] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5076.151158] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5085.482861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5090.786584] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5090.787188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5090.787876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5090.787882] Lustre: Skipped 3 previous similar messages [ 5090.790261] Lustre: Skipped 4 previous similar messages [ 5093.880745] Lustre: server umount lustre-OST0000 complete [ 5101.208190] Lustre: server umount lustre-MDT0000 complete [ 5102.392427] LustreError: 361416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680912 with bad export cookie 7486314336015170049 [ 5102.394868] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5102.395308] LustreError: 361416:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5102.549603] Lustre: server umount lustre-MDT0001 complete [ 5106.065418] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 5107.097245] Key type lgssc unregistered [ 5107.240390] LNet: 364591:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5107.242553] LNetError: 364591:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5107.255297] LNet: Removed LNI 192.168.201.118@tcp [ 5107.544340] Key type .llcrypt unregistered [ 5107.545320] Key type ._llcrypt unregistered [ 5116.746789] Key type ._llcrypt registered [ 5116.747720] Key type .llcrypt registered [ 5116.786428] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5117.253915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5117.296754] alg: No test for adler32 (adler32-zlib) [ 5118.173579] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 5118.266351] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 5119.855165] Key type lgssc registered [ 5120.326446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5123.152986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5123.159659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5124.342118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5125.718263] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5128.856529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5128.992329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5130.640100] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5132.928574] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5134.331162] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5136.559716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5136.698300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5138.541252] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5140.290241] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5151.021238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5159.903949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5159.907618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5159.913065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5161.441616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5161.441747] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5161.443319] Lustre: Skipped 1 previous similar message [ 5161.446290] Lustre: Skipped 1 previous similar message [ 5165.952209] LustreError: 368336:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5165.973190] Lustre: server umount lustre-MDT0000 complete [ 5166.561154] LustreError: 366778:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5166.565190] LustreError: 366778:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 4 previous similar messages [ 5167.103717] LustreError: 368237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756680976 with bad export cookie 7641814790054364748 [ 5167.107242] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5167.108054] LustreError: 368237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5167.167095] LustreError: 368538:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5167.168847] LustreError: 368538:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5167.245119] Lustre: server umount lustre-MDT0001 complete [ 5178.673779] LustreError: 368738:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5178.675431] LustreError: 368738:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5178.744515] Lustre: server umount lustre-OST0000 complete [ 5182.979923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5183.137840] LustreError: 369468:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5183.162861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5184.361728] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5187.056255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5187.175382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5188.476218] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5190.119911] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5191.170534] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5193.328580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5195.319881] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5196.970195] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5207.779572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5222.879840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5222.881754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5222.884968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5225.407187] LustreError: 371837:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5225.409640] LustreError: 371837:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5225.437886] Lustre: server umount lustre-MDT0000 complete [ 5226.659663] LustreError: 369449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756681036 with bad export cookie 7641814790054365847 [ 5226.663830] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5226.814894] Lustre: server umount lustre-MDT0001 complete [ 5238.257622] LustreError: 372241:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5238.259745] LustreError: 372241:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5238.334687] Lustre: server umount lustre-OST0000 complete [ 5243.125599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5243.281150] LustreError: 373068:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5243.302909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5243.304568] Lustre: Skipped 1 previous similar message [ 5244.537862] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5247.327266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5248.721918] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5250.535656] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5251.730162] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5253.932321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5255.894391] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5257.650622] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5268.395520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5278.688631] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5278.689917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5278.690575] LustreError: Skipped 1 previous similar message [ 5278.690666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5278.690670] Lustre: Skipped 3 previous similar messages [ 5278.693477] Lustre: Skipped 3 previous similar messages [ 5280.685582] LustreError: 375336:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5280.687206] LustreError: 375336:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5280.753886] Lustre: server umount lustre-OST0000 complete [ 5283.295939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5283.299595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5283.305800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5283.308200] Lustre: Skipped 1 previous similar message [ 5288.085655] Lustre: server umount lustre-MDT0000 complete [ 5288.928531] LustreError: 373067:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5288.932227] LustreError: 373067:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 5289.292558] LustreError: 373049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756681099 with bad export cookie 7641814790054366855 [ 5289.295438] LustreError: 373049:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5289.297282] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5289.461244] Lustre: server umount lustre-MDT0001 complete [ 5293.003487] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 5294.068445] Key type lgssc unregistered [ 5294.210551] LNet: 376270:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5294.213334] LNetError: 376270:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5294.220577] LNet: Removed LNI 192.168.201.118@tcp [ 5294.561392] Key type .llcrypt unregistered [ 5294.562423] Key type ._llcrypt unregistered [ 5304.064435] Key type ._llcrypt registered [ 5304.065442] Key type .llcrypt registered [ 5304.108329] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5304.493136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5304.538681] alg: No test for adler32 (adler32-zlib) [ 5305.399355] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 5305.491858] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 5307.079134] Key type lgssc registered [ 5307.470564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5310.147648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5310.152937] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5310.154978] LustreError: 377667:0:(osd_handler.c:8487:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5310.165975] LustreError: 377667:0:(obd_config.c:782:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5310.167739] LustreError: 377667:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5310.169510] LustreError: 377667:0:(tgt_mount.c:2428:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5310.171511] LustreError: 377667:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5312.613659] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5312.615769] LustreError: 378064:0:(osd_handler.c:8487:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5312.623864] LustreError: 378064:0:(obd_config.c:782:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5312.625609] LustreError: 378064:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5312.627397] LustreError: 378064:0:(tgt_mount.c:2428:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5312.629356] LustreError: 378064:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5319.192909] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 5320.323854] Key type lgssc unregistered [ 5320.469493] LNet: 379048:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5320.473247] LNetError: 379048:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5320.483380] LNet: Removed LNI 192.168.201.118@tcp [ 5320.783481] Key type .llcrypt unregistered [ 5320.784545] Key type ._llcrypt unregistered [ 5326.497215] Key type ._llcrypt registered [ 5326.498211] Key type .llcrypt registered [ 5326.533270] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 5332.910894] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5333.270575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5333.315299] alg: No test for adler32 (adler32-zlib) [ 5334.190093] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 5334.294721] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 5335.895143] Key type lgssc registered [ 5336.204400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5338.903864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5340.998132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5343.015902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5344.980984] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5347.244333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5347.267115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5347.271935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5348.362149] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5348.376563] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5348.417179] Lustre: lustre-MDT0000: new disk, initializing [ 5348.443449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5348.450229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5349.819124] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5353.902892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5353.927764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5353.957735] Lustre: 382150:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5353.981415] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5353.982936] Lustre: Skipped 1 previous similar message [ 5354.018809] Lustre: lustre-MDT0001: new disk, initializing [ 5354.056358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5354.069110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5354.074180] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5355.313133] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5358.290310] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5359.411919] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5361.604568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5361.626541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5361.732356] Lustre: lustre-OST0000: new disk, initializing [ 5361.735381] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5361.768461] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5362.623198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5362.628255] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5362.675129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5363.689446] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5366.643339] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5367.944518] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5368.009151] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5369.141785] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5369.202681] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5372.897552] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5372.897577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5372.900611] LustreError: Skipped 1 previous similar message [ 5372.906108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5372.908787] Lustre: Skipped 1 previous similar message [ 5378.018498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5378.020075] Lustre: Skipped 1 previous similar message [ 5383.137419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5383.139063] Lustre: Skipped 1 previous similar message [ 5384.159121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5384.176079] LustreError: 385090:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5384.242489] Lustre: server umount lustre-OST0000 complete [ 5388.256928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5388.257734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5388.260159] Lustre: Skipped 2 previous similar messages [ 5391.549612] LustreError: 385292:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5391.552836] LustreError: 385292:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5391.584308] Lustre: server umount lustre-MDT0000 complete [ 5392.784679] LustreError: 382143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756681202 with bad export cookie 6912008178483600429 [ 5392.789236] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5392.861715] LustreError: 385493:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5392.863509] LustreError: 385493:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5392.955667] Lustre: server umount lustre-MDT0001 complete [ 5395.539735] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5396.042552] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 19:00:04 (1756681204) [ 5398.166112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5398.344434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5399.515703] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5402.207796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5403.673254] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5405.353569] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5406.415480] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5408.544010] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5408.656860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5408.659269] Lustre: Skipped 1 previous similar message [ 5410.356621] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5412.038501] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5414.302265] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5414.328083] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5414.384633] Lustre: lustre-OST0001: new disk, initializing [ 5414.386566] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5416.309163] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5419.344907] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5422.071597] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5422.073388] Lustre: Skipped 1 previous similar message [ 5427.179629] LustreError: 389399:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5427.181242] LustreError: 389399:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5427.193612] Lustre: server umount lustre-OST0001 complete [ 5432.292860] LustreError: 387806:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5432.296668] LustreError: 387806:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 5437.411935] LustreError: 387806:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5437.419069] LustreError: 387806:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 5440.480096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5440.482286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5440.486159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5440.487683] Lustre: Skipped 4 previous similar messages [ 5442.532700] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5442.532768] LustreError: 388001:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5442.539879] LustreError: 388001:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 5445.166073] LustreError: 389634:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5445.168013] LustreError: 389634:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5445.239405] Lustre: server umount lustre-OST0000 complete [ 5447.650056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5452.637047] Lustre: server umount lustre-MDT0000 complete [ 5452.768240] LustreError: 387660:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5452.772139] LustreError: 387660:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 5453.795582] LustreError: 386192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756681263 with bad export cookie 6912008178483601297 [ 5453.800026] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5453.801545] LustreError: 386192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5468.127084] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5468.162668] LustreError: 390048:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5468.165858] LustreError: 390048:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5468.263229] Lustre: server umount lustre-MDT0001 complete [ 5471.768788] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 5472.724435] Key type lgssc unregistered [ 5472.858420] LNet: 390581:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5472.862291] LNetError: 390581:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5472.873328] LNet: Removed LNI 192.168.201.118@tcp [ 5473.157375] Key type .llcrypt unregistered [ 5473.158508] Key type ._llcrypt unregistered [ 5483.402969] Key type ._llcrypt registered [ 5483.403844] Key type .llcrypt registered [ 5483.436832] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 5489.366457] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5489.911247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5489.922324] alg: No test for adler32 (adler32-zlib) [ 5490.764256] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 5490.841963] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 5492.423179] Key type lgssc registered [ 5492.764308] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5495.550272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5497.597189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5499.489754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5501.433749] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5503.649410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5503.678753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5503.684991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5504.786678] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5504.799668] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5504.837265] Lustre: lustre-MDT0000: new disk, initializing [ 5504.864427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5504.870797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5506.102681] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5510.068553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5510.094931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5510.128928] Lustre: 394502:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5510.159882] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5510.162350] Lustre: Skipped 1 previous similar message [ 5510.201611] Lustre: lustre-MDT0001: new disk, initializing [ 5510.237483] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5510.248496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5510.252978] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5511.494063] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5514.548454] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5515.738797] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5517.882511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5517.905795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5518.002917] Lustre: lustre-OST0000: new disk, initializing [ 5518.004689] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5518.029176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5519.700977] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5522.555824] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5522.938051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5522.940908] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5522.978070] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5523.785450] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5523.849532] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5524.968991] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5525.030311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5528.033594] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5528.033799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5528.037934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5528.038976] LustreError: Skipped 1 previous similar message [ 5531.637087] LustreError: 397439:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5531.701348] Lustre: server umount lustre-OST0000 complete [ 5533.152689] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5533.153673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5533.155848] Lustre: Skipped 2 previous similar messages [ 5533.158789] Lustre: Skipped 2 previous similar messages [ 5538.272108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5538.273779] Lustre: Skipped 1 previous similar message [ 5539.004610] LustreError: 397640:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5539.007833] LustreError: 397640:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5539.036696] Lustre: server umount lustre-MDT0000 complete [ 5540.198437] LustreError: 394493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756681349 with bad export cookie 14000233947042294822 [ 5540.200713] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5540.201578] LustreError: 394493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5540.270851] LustreError: 397843:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5540.272513] LustreError: 397843:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5540.357380] Lustre: server umount lustre-MDT0001 complete [ 5543.008870] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 19:02:31 (1756681351) [ 5545.817110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5548.428067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5548.571750] Lustre: lustre-MDT0000: new disk, initializing [ 5548.607265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5549.861969] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5552.532561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5552.658260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5553.923907] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5555.537238] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5556.609243] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5557.209248] LustreError: 399137:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5557.212201] LustreError: 399137:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5557.214598] LustreError: 399137:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 5557.728364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5557.728579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5557.728919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5557.733030] Lustre: Skipped 1 previous similar message [ 5562.848390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5562.850140] Lustre: Skipped 2 previous similar messages [ 5563.321551] LustreError: 400447:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5563.324125] LustreError: 400447:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5563.347757] Lustre: server umount lustre-MDT0000 complete [ 5564.545245] LustreError: 399132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756681374 with bad export cookie 14000233947042295690 [ 5564.549421] LustreError: 399132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5564.551098] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5564.699966] Lustre: server umount lustre-MDT0001 complete [ 5567.542540] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 19:02:56 (1756681376) [ 5569.616779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5569.775300] LustreError: 401368:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5569.780422] LustreError: 401368:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 5569.797643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5570.983905] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5573.627317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5574.939778] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5576.560985] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5577.745471] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5579.863446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5581.770603] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5583.481134] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5587.042896] Lustre: Permanently deactivating lustre-OST0000 [ 5587.047817] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5593.071497] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5593.974714] LustreError: 404058:0:(mgs_llog.c:1937:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5593.977523] LustreError: 404058:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 5593.979373] LustreError: 404058:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5594.570742] LustreError: 404106:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 5594.574103] LustreError: 404106:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5601.261160] LustreError: 404203:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5601.264463] LustreError: 404203:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5601.343860] Lustre: server umount lustre-OST0000 complete [ 5602.270894] LustreError: 404355:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 5602.273116] LustreError: 404355:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5604.320027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5604.322055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5604.325703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5613.026627] LustreError: 404452:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5613.028312] LustreError: 404452:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5613.050921] Lustre: server umount lustre-MDT0000 complete [ 5614.185405] LustreError: 401349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756681423 with bad export cookie 14000233947042296208 [ 5614.188967] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5614.190644] LustreError: 401349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5623.854727] Lustre: server umount lustre-MDT0001 complete [ 5626.416746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5627.573222] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5628.382374] LustreError: 405495:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5630.252469] Lustre: 405833:0:(mgs_llog.c:1698:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.118@tcp [ 5630.853630] LustreError: 405930:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5630.855423] LustreError: 405930:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5630.876977] Lustre: server umount MGS complete [ 5634.341070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5634.500811] LustreError: 406559:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5634.505210] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5634.508323] Lustre: Skipped 1 previous similar message [ 5634.531079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5634.534844] Lustre: Skipped 2 previous similar messages [ 5635.782526] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5638.486341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5638.593497] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5639.898978] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5641.598944] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5642.722628] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5643.326835] Lustre: Permanently reactivating lustre-OST0000 [ 5643.331505] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5643.333500] Lustre: Skipped 2 previous similar messages [ 5668.832159] LustreError: 408113:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5670.914171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5671.035976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5671.037849] Lustre: Skipped 1 previous similar message [ 5672.728892] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5674.285387] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5688.803070] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5688.804781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5688.805305] LustreError: Skipped 1 previous similar message [ 5688.805609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5688.805617] Lustre: Skipped 4 previous similar messages [ 5688.808228] Lustre: Skipped 2 previous similar messages [ 5691.886109] LustreError: 409942:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5691.887681] LustreError: 409942:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5691.958458] Lustre: server umount lustre-OST0000 complete [ 5699.218496] Lustre: server umount lustre-MDT0000 complete [ 5700.319370] LustreError: 406539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756681510 with bad export cookie 14000233947042297153 [ 5700.324031] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5700.324978] LustreError: 406539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5700.469392] Lustre: server umount lustre-MDT0001 complete [ 5703.677139] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 5704.636576] Key type lgssc unregistered [ 5704.763325] LNet: 410878:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5704.765678] LNetError: 410878:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5704.772284] LNet: Removed LNI 192.168.201.118@tcp [ 5705.038571] Key type .llcrypt unregistered [ 5705.039615] Key type ._llcrypt unregistered [ 5710.341975] Key type ._llcrypt registered [ 5710.342961] Key type .llcrypt registered [ 5710.375349] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 5716.009044] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5716.413945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5716.437351] alg: No test for adler32 (adler32-zlib) [ 5717.287567] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 5717.366157] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 5718.943156] Key type lgssc registered [ 5719.276453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5722.096691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5724.152716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5726.099429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5728.132643] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5730.347133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5730.363818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5730.368419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5731.459212] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5731.469799] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5731.504121] Lustre: lustre-MDT0000: new disk, initializing [ 5731.527699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5731.533599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5732.695333] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5736.402648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5736.424717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5736.454061] Lustre: 413982:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5736.477672] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5736.479131] Lustre: Skipped 1 previous similar message [ 5736.516018] Lustre: lustre-MDT0001: new disk, initializing [ 5736.549176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5736.560114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5736.564097] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5737.692347] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5740.573476] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5741.658486] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5743.601598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5743.631942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5743.728539] Lustre: lustre-OST0000: new disk, initializing [ 5743.730323] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5743.756700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5744.570950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5744.573455] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5744.612344] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5745.387343] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5748.282913] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5749.429909] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5749.494350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5750.589858] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5750.647277] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5754.851110] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5754.851290] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5754.852898] LustreError: Skipped 1 previous similar message [ 5754.856155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5754.858034] Lustre: Skipped 1 previous similar message [ 5757.292126] LustreError: 416921:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5757.350412] Lustre: server umount lustre-OST0000 complete [ 5759.969290] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5759.969291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5759.969310] Lustre: Skipped 1 previous similar message [ 5759.971840] Lustre: Skipped 1 previous similar message [ 5759.972248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5759.980228] Lustre: Skipped 1 previous similar message [ 5764.604984] LustreError: 417124:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5764.606443] LustreError: 417124:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5764.637766] Lustre: server umount lustre-MDT0000 complete [ 5765.711359] LustreError: 414761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756681575 with bad export cookie 6299603182285092989 [ 5765.715288] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5765.716403] LustreError: 414761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5765.786959] LustreError: 417326:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5765.790113] LustreError: 417326:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5765.877442] Lustre: server umount lustre-MDT0001 complete [ 5768.338521] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5768.825500] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 19:06:17 (1756681577) [ 5770.982199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5771.119264] LustreError: 418045:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5771.123184] LustreError: 418045:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 5771.141282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5772.324512] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5775.401386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5775.522704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5776.743292] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5778.358533] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5779.476832] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5781.678554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5783.594670] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5785.284643] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5785.912444] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5796.815437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5801.441704] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5801.443399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5801.444140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5801.449444] Lustre: Skipped 1 previous similar message [ 5804.268902] LustreError: 420502:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5804.270470] LustreError: 420502:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5804.337100] Lustre: server umount lustre-OST0000 complete [ 5806.561183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5806.561252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5806.564830] LustreError: Skipped 1 previous similar message [ 5806.568111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5806.572531] Lustre: Skipped 2 previous similar messages [ 5811.643262] LustreError: 420704:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5811.645014] LustreError: 420704:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5811.669974] Lustre: server umount lustre-MDT0000 complete [ 5811.680570] LustreError: 418044:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5811.686481] LustreError: 418044:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 5812.858233] LustreError: 419662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756681622 with bad export cookie 6299603182285093857 [ 5812.861202] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5812.861506] LustreError: 419662:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5813.018615] Lustre: server umount lustre-MDT0001 complete [ 5816.526696] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 5817.563169] Key type lgssc unregistered [ 5817.701304] LNet: 421437:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5817.703263] LNetError: 421437:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5817.714376] LNet: Removed LNI 192.168.201.118@tcp [ 5818.029832] Key type .llcrypt unregistered [ 5818.030810] Key type ._llcrypt unregistered [ 5825.928157] Key type ._llcrypt registered [ 5825.929246] Key type .llcrypt registered [ 5825.963947] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 5826.069416] Key type .llcrypt unregistered [ 5826.071275] Key type ._llcrypt unregistered [ 5832.401614] Key type ._llcrypt registered [ 5832.402552] Key type .llcrypt registered [ 5832.439918] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5832.947720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5832.954920] alg: No test for adler32 (adler32-zlib) [ 5833.819355] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 5833.902525] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 5835.479187] Key type lgssc registered [ 5835.855658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5838.335688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5838.342399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5839.514925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5840.702293] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5843.436745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5843.544608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5845.261377] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5846.830517] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5848.545405] LustreError: 423585:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5848.549383] LustreError: 423585:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 4 previous similar messages [ 5848.934463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5849.055545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5850.088360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5850.271823] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5854.176447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5854.178669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5854.182252] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5855.202501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5855.203435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5855.207370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5858.990254] LustreError: 425638:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 5859.045500] Lustre: server umount lustre-OST0000 complete [ 5860.320510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5860.321233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5860.323641] Lustre: Skipped 1 previous similar message [ 5860.325224] Lustre: Skipped 1 previous similar message [ 5865.440688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5865.442476] Lustre: Skipped 1 previous similar message [ 5866.299997] LustreError: 425840:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5866.301600] LustreError: 425840:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5866.333993] Lustre: server umount lustre-MDT0000 complete [ 5867.487301] LustreError: 423565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756681677 with bad export cookie 17708470926712454839 [ 5867.492057] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5867.561418] LustreError: 426042:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5867.563113] LustreError: 426042:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5867.652791] Lustre: server umount lustre-MDT0001 complete [ 5871.090989] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 5872.325913] Key type lgssc unregistered [ 5872.498799] LNet: 426573:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5872.503298] LNetError: 426573:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5872.514699] LNet: Removed LNI 192.168.201.118@tcp [ 5872.922777] Key type .llcrypt unregistered [ 5872.924573] Key type ._llcrypt unregistered [ 5881.979755] Key type ._llcrypt registered [ 5881.980892] Key type .llcrypt registered [ 5882.020142] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5882.399469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5882.470178] alg: No test for adler32 (adler32-zlib) [ 5883.316466] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 5883.397493] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 5884.975119] Key type lgssc registered [ 5885.300615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5887.665803] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5887.671576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5904.159245] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5910.303146] LustreError: 427919:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.118@tcp: failed processing log, type 4: rc = -110 [ 5941.023119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5942.718111] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5944.249332] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5946.268845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5946.435106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5946.460166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5947.537292] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5950.083805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5950.204256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5951.324103] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5952.826875] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5953.873171] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5960.672507] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5960.674649] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5960.678577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5961.696509] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5961.702170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5961.710645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5964.770309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5964.772335] Lustre: Skipped 1 previous similar message [ 5966.188096] LustreError: 430200:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 5966.243888] Lustre: server umount lustre-OST0000 complete [ 5969.888806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5969.893072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5969.894591] Lustre: Skipped 1 previous similar message [ 5973.435245] LustreError: 430401:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 5973.437161] LustreError: 430401:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5973.462235] Lustre: server umount lustre-MDT0000 complete [ 5974.525050] LustreError: 428873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756681784 with bad export cookie 5177616634682111211 [ 5974.527789] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5974.528101] LustreError: 428873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5974.592184] LustreError: 430602:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5974.594365] LustreError: 430602:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5974.673502] Lustre: server umount lustre-MDT0001 complete [ 5978.097953] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 5979.120989] Key type lgssc unregistered [ 5979.253443] LNet: 431133:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5979.255378] LNetError: 431133:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5979.266378] LNet: Removed LNI 192.168.201.118@tcp [ 5979.577858] Key type .llcrypt unregistered [ 5979.578843] Key type ._llcrypt unregistered [ 5988.522472] Key type ._llcrypt registered [ 5988.523451] Key type .llcrypt registered [ 5988.558806] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 5989.017036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5989.062137] alg: No test for adler32 (adler32-zlib) [ 5989.903775] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 5989.981673] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 5991.559176] Key type lgssc registered [ 5991.894432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5994.157629] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5994.163565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5995.311880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5996.475367] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 5999.089174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5999.203269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6000.399750] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6001.986618] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6003.129940] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6005.212784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6005.323504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6006.374905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6007.019438] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6008.570527] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6013.922171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6013.923587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6013.925503] Lustre: Skipped 2 previous similar messages [ 6013.927291] Lustre: Skipped 1 previous similar message [ 6014.862552] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.18@tcp (stopping) [ 6016.443132] LustreError: 434748:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6016.469874] Lustre: server umount lustre-MDT0000 complete [ 6017.128781] LustreError: 432488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.18@tcp arrived at 1756681826 with bad export cookie 12317481874195743252 [ 6017.132263] LustreError: 432488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6017.703043] LustreError: 432488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756681827 with bad export cookie 12317481874195742398 [ 6017.706529] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6019.040168] LustreError: 432506:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6019.040502] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6019.040702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6019.041485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6019.044456] LustreError: 432506:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 3 previous similar messages [ 6022.047870] LustreError: 432506:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6022.048209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6022.052128] LustreError: 432506:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 6023.789090] LustreError: 434951:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6023.790883] LustreError: 434951:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6023.851729] Lustre: server umount lustre-OST0000 complete [ 6025.868398] LustreError: 435304:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6025.870168] LustreError: 435304:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6025.957913] Lustre: server umount lustre-MDT0001 complete [ 6029.162546] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 6030.094755] Key type lgssc unregistered [ 6030.227348] LNet: 435835:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6030.229502] LNetError: 435835:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6030.238324] LNet: Removed LNI 192.168.201.118@tcp [ 6030.516544] Key type .llcrypt unregistered [ 6030.517736] Key type ._llcrypt unregistered [ 6038.669217] Key type ._llcrypt registered [ 6038.670289] Key type .llcrypt registered [ 6038.702399] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6039.008210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6039.045062] alg: No test for adler32 (adler32-zlib) [ 6039.889540] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 6039.966386] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 6041.543119] Key type lgssc registered [ 6041.883863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6044.485903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6044.492682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6045.655724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6046.921451] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6050.083060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6050.213241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6051.414311] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6053.346404] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6054.425094] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6056.586401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6056.704064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6058.997015] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6060.706419] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6071.082206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6075.871851] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6075.873734] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6075.877151] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6076.385143] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6076.386669] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6076.386982] Lustre: Skipped 1 previous similar message [ 6078.074943] LustreError: 439475:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6078.094373] Lustre: server umount lustre-MDT0001 complete [ 6081.506570] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6081.507456] LustreError: 437935:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6081.507505] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6081.508415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6081.515778] LustreError: 437935:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 6085.678140] LustreError: 439680:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6085.679850] LustreError: 439680:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6085.747029] Lustre: server umount lustre-OST0000 complete [ 6086.626405] LustreError: 437935:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6086.940734] LustreError: 439881:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6086.942387] LustreError: 439881:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6087.048996] Lustre: server umount lustre-MDT0000 complete [ 6091.058811] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 6092.050257] Key type lgssc unregistered [ 6092.182451] LNet: 440564:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6092.184327] LNetError: 440564:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6092.194391] LNet: Removed LNI 192.168.201.118@tcp [ 6092.485477] Key type .llcrypt unregistered [ 6092.486404] Key type ._llcrypt unregistered [ 6099.635156] Key type ._llcrypt registered [ 6099.636098] Key type .llcrypt registered [ 6099.673626] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 6105.096238] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6105.524266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6105.561249] alg: No test for adler32 (adler32-zlib) [ 6106.415554] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 6106.503756] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 6108.087121] Key type lgssc registered [ 6108.449477] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6111.225946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6113.064801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6114.756161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6116.530586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6119.089761] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6122.432682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6124.605432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6124.623389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6124.629214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6125.710104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6125.720788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6125.755280] Lustre: lustre-MDT0000: new disk, initializing [ 6125.777224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6125.782673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6126.938798] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6130.159562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6132.090085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6132.111827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6132.197463] Lustre: lustre-OST0000: new disk, initializing [ 6132.199256] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6132.200697] Lustre: Skipped 1 previous similar message [ 6132.221414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6133.500249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6133.502705] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6133.511095] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6133.847866] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6136.623293] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6139.575013] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6141.529982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6141.555926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6141.585779] Lustre: 444513:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6141.609413] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6141.644398] Lustre: lustre-MDT0001: new disk, initializing [ 6141.682238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6141.695105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6141.697873] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6142.790612] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6145.967300] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6146.034436] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6147.166265] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6147.235257] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6150.031224] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.18@tcp (stopping) [ 6152.672704] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6152.673680] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6152.675496] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6152.680793] Lustre: Skipped 1 previous similar message [ 6154.747607] LustreError: 447933:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6154.768819] Lustre: server umount lustre-MDT0001 complete [ 6162.913849] LustreError: 445526:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6162.919024] LustreError: 445526:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 5 previous similar messages [ 6168.033922] LustreError: 446750:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6168.034268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6168.041305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6168.042819] Lustre: Skipped 1 previous similar message [ 6173.153591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6173.154211] LustreError: 446750:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6178.272866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6178.274167] LustreError: 446750:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6181.343099] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6181.364520] LustreError: 448205:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 6181.366155] LustreError: 448205:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6181.429476] Lustre: server umount lustre-OST0000 complete [ 6188.868361] LustreError: 448407:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6188.870841] LustreError: 448407:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6188.983619] Lustre: server umount lustre-MDT0000 complete [ 6193.946406] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 6195.005564] Key type lgssc unregistered [ 6195.145391] LNet: 449092:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6195.147498] LNetError: 449092:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6195.156394] LNet: Removed LNI 192.168.201.118@tcp [ 6195.467883] Key type .llcrypt unregistered [ 6195.468877] Key type ._llcrypt unregistered [ 6212.755432] Key type ._llcrypt registered [ 6212.756588] Key type .llcrypt registered [ 6212.805978] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 6212.925314] Key type .llcrypt unregistered [ 6212.927488] Key type ._llcrypt unregistered [ 6221.041424] Key type ._llcrypt registered [ 6221.044239] Key type .llcrypt registered [ 6221.100649] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6221.518584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6221.550102] alg: No test for adler32 (adler32-zlib) [ 6222.401029] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 6222.485897] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 6224.063122] Key type lgssc registered [ 6224.609967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6226.995201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6229.277419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6231.730571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6234.587768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6234.618684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6234.625532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6235.727615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6235.739609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6235.779862] Lustre: lustre-MDT0000: new disk, initializing [ 6235.805589] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6235.811620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6237.007296] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6241.062687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6241.088742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6241.119388] Lustre: 452409:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6241.144935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6241.146523] Lustre: Skipped 1 previous similar message [ 6241.180816] Lustre: lustre-MDT0001: new disk, initializing [ 6241.225370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6241.238849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6241.242558] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6242.698686] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6246.375112] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6248.163464] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6251.231826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6251.284473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6251.458921] Lustre: lustre-OST0000: new disk, initializing [ 6251.461834] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6251.518949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6254.288827] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6257.968875] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6261.787065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6261.793681] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6261.859493] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6266.855070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6266.858290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6266.862354] Lustre: Skipped 2 previous similar messages [ 6266.865955] Lustre: Skipped 1 previous similar message [ 6266.871474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6271.891344] LustreError: 454853:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6271.942976] Lustre: server umount lustre-MDT0000 complete [ 6273.833748] LustreError: 452401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756682083 with bad export cookie 6076003254099030757 [ 6273.841003] LustreError: 452401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6273.843745] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6273.933194] LustreError: 455057:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6273.937023] LustreError: 455057:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6274.052934] Lustre: server umount lustre-MDT0001 complete [ 6285.617368] LustreError: 455257:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6285.619091] LustreError: 455257:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6285.695912] Lustre: server umount lustre-OST0000 complete [ 6289.976266] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 19:14:58 (1756682098) [ 6290.300907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6293.262039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6293.519126] LustreError: 456085:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6293.527158] LustreError: 456085:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 4 previous similar messages [ 6293.556422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6295.080404] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6298.486549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6298.592895] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6298.596642] Lustre: Skipped 1 previous similar message [ 6298.687801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6300.179276] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6302.190929] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6303.901191] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6306.149166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6306.189543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6306.303160] Lustre: Found index 0 for lustre-OST0000, updating log [ 6309.318924] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6310.979973] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6319.715338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6324.195248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6324.199658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6324.207682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6329.268163] LustreError: 458361:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6329.272115] LustreError: 458361:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6329.361796] Lustre: server umount lustre-OST0000 complete [ 6330.231269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6331.939667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6331.946689] Lustre: Skipped 1 previous similar message [ 6331.950260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6331.953924] Lustre: Skipped 4 previous similar messages [ 6336.994212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6336.998705] Lustre: Skipped 1 previous similar message [ 6345.183196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6345.236540] LustreError: 458615:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6345.240710] LustreError: 458615:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6345.278092] Lustre: server umount lustre-MDT0000 complete [ 6347.090346] LustreError: 456064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756682156 with bad export cookie 6076003254099032122 [ 6347.096913] LustreError: 456064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6347.097952] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6347.324195] Lustre: server umount lustre-MDT0001 complete [ 6351.374435] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 19:15:59 (1756682159) [ 6351.893182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6352.415057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6352.937531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6353.430761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6353.952176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6354.446100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6354.956380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6355.470085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6355.985335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6356.514644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6357.053883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6357.560685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6358.085827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6358.593847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6359.122169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6359.636939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6360.145853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6360.649220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6361.174140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6361.686162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6362.186661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6362.711396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6363.202541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6363.680958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6364.197451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6364.721655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6365.238475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6365.767991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6366.285516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6366.804225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6367.310872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6367.836433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6368.362053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6368.889112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6369.433078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6369.947684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6370.469630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6370.995197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6371.509752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6372.049095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6372.535577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6373.051994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6373.548985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6374.053737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6374.568763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6375.083142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6375.609251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6376.104079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6376.627366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6377.157465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6377.692103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6381.608391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6381.852842] LustreError: 462206:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6381.884398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6381.886555] Lustre: Skipped 1 previous similar message [ 6383.802808] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6387.168566] LustreError: 462206:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6387.520907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6388.971413] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6390.972434] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6392.075305] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6394.385442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6394.414356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6394.492909] Lustre: Found index 0 for lustre-OST0000, updating log [ 6396.334764] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6398.259025] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6403.045061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6403.045298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6403.048235] Lustre: Skipped 1 previous similar message [ 6403.051755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6403.053207] Lustre: Skipped 2 previous similar messages [ 6404.974211] LustreError: 464522:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6404.977912] LustreError: 464522:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6405.066648] Lustre: server umount lustre-OST0000 complete [ 6407.394199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6407.399533] Lustre: Skipped 1 previous similar message [ 6412.514823] LustreError: 462205:0:(ldlm_lib.c:1132:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6412.527641] LustreError: 462205:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 1 previous similar message [ 6412.542493] Lustre: server umount lustre-MDT0000 complete [ 6413.925651] LustreError: 462187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756682223 with bad export cookie 6076003254099033193 [ 6413.928808] LustreError: 462187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6413.929247] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6414.077443] Lustre: server umount lustre-MDT0001 complete [ 6414.927557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6420.145508] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 6422.871082] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6426.611076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6429.566880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6432.453524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6434.806995] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6437.839584] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 19:17:26 (1756682246) [ 6444.368421] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 6445.674206] Key type lgssc unregistered [ 6456.096326] LNet: 468486:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6456.100871] LNetError: 468486:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6567.719968] LNet: Removed LNI 192.168.201.118@tcp [ 6568.208812] Key type .llcrypt unregistered [ 6568.210942] Key type ._llcrypt unregistered [ 6576.949502] Key type ._llcrypt registered [ 6576.951472] Key type .llcrypt registered [ 6577.017743] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6577.456072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6577.518261] alg: No test for adler32 (adler32-zlib) [ 6578.425818] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 6578.526634] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 6580.127200] Key type lgssc registered [ 6580.784948] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6583.183894] LNet: Added LNI 192.168.201.118@tcp100 [8/256/0/180] [ 6584.003377] Lustre: DEBUG MARKER: SKIP: conf-sanity test_73c unable to configure net [ 6591.833926] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 6592.987735] Key type lgssc unregistered [ 6593.166885] LNet: 470351:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6593.170901] LNetError: 470351:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6593.174061] LNet: Removed LNI 192.168.201.118@tcp [ 6593.183549] LNet: Removed LNI 192.168.201.118@tcp100 [ 6593.579647] Key type .llcrypt unregistered [ 6593.581811] Key type ._llcrypt unregistered [ 6604.516878] Key type ._llcrypt registered [ 6604.518941] Key type .llcrypt registered [ 6604.588713] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 6612.710333] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6613.114820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6613.124134] alg: No test for adler32 (adler32-zlib) [ 6614.047947] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 6614.163851] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 6615.767174] Key type lgssc registered [ 6616.403305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6621.288052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6624.965909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6628.162590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6631.417260] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6634.071721] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 19:20:42 (1756682442) [ 6639.922390] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 6641.040315] Key type lgssc unregistered [ 6641.187432] LNet: 474362:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6641.190096] LNetError: 474362:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6641.199289] LNet: Removed LNI 192.168.201.118@tcp [ 6641.502378] Key type .llcrypt unregistered [ 6641.503850] Key type ._llcrypt unregistered [ 6648.395612] Key type ._llcrypt registered [ 6648.396678] Key type .llcrypt registered [ 6648.432284] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6648.741243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6648.794575] alg: No test for adler32 (adler32-zlib) [ 6649.644637] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 6649.723042] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 6651.295206] Key type lgssc registered [ 6651.614214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6654.535608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6654.558900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6654.565126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6655.661159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6655.672313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6655.712196] Lustre: lustre-MDT0000: new disk, initializing [ 6655.738343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6655.744571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6656.952228] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6661.041875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6661.076237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6661.110540] Lustre: 475533:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6661.137362] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6661.138946] Lustre: Skipped 1 previous similar message [ 6661.178772] Lustre: lustre-MDT0001: new disk, initializing [ 6661.221760] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6661.236645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6661.239857] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6662.493880] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6665.539910] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6667.223719] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6670.694432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6670.731093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6670.876431] Lustre: lustre-OST0000: new disk, initializing [ 6670.879091] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6670.928407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6673.733338] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6677.826495] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6678.556444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6678.562795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6678.616132] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6679.374436] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing lnet_if_list [ 6681.712534] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 6681.732936] LNet: Added LNI 192.168.201.118@tcp42 [8/256/0/180] [ 6693.862867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6693.863549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6693.867239] LustreError: Skipped 1 previous similar message [ 6693.875346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6693.880638] Lustre: Skipped 1 previous similar message [ 6696.562776] LustreError: 478917:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6696.666564] Lustre: server umount lustre-OST0000 complete [ 6698.978520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6698.979734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6698.986946] Lustre: Skipped 2 previous similar messages [ 6704.097955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6704.101917] Lustre: Skipped 2 previous similar messages [ 6704.654617] LustreError: 479119:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6704.658271] LustreError: 479119:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6704.704275] Lustre: server umount lustre-MDT0000 complete [ 6706.588429] LustreError: 475523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756682516 with bad export cookie 10438019208063521779 [ 6706.595232] LustreError: 475523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6706.595996] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6706.696730] LustreError: 479320:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6706.699706] LustreError: 479320:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6706.807746] Lustre: server umount lustre-MDT0001 complete [ 6712.096865] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 6713.796855] Key type lgssc unregistered [ 6714.029380] LNet: 479859:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6714.035321] LNetError: 479859:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6714.040174] LNet: Removed LNI 192.168.201.118@tcp [ 6714.048780] LNet: Removed LNI 192.168.201.118@tcp42 [ 6714.498577] Key type .llcrypt unregistered [ 6714.500593] Key type ._llcrypt unregistered [ 6724.313534] Key type ._llcrypt registered [ 6724.315264] Key type .llcrypt registered [ 6724.383905] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6724.960786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6724.988162] alg: No test for adler32 (adler32-zlib) [ 6725.899373] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 6726.027502] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 6727.631201] Key type lgssc registered [ 6728.157650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6731.458236] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6731.465410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6732.653108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6733.899641] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6736.849893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6736.973808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6738.215770] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6739.840728] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6740.990794] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6743.042189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6743.149296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6745.112297] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6746.671988] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6765.923372] loop0: detected capacity change from 0 to 204800000 [ 6765.986704] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6765.990528] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6765.994630] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6765.998552] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6766.019981] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6766.038699] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6766.052288] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6766.074604] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6766.100588] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6766.133130] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6768.052734] loop0: detected capacity change from 0 to 204800000 [ 6768.126958] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6768.178395] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6770.224947] loop0: detected capacity change from 0 to 204800000 [ 6772.401878] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6772.430805] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6772.478660] Lustre: fs15246-OST0000: new disk, initializing [ 6772.483511] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 6772.515976] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6774.801517] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6779.111326] loop1: detected capacity change from 0 to 204800000 [ 6781.146774] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6781.176949] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6781.214928] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 6781.237886] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 6781.272133] Lustre: fs15246-MDT0000: new disk, initializing [ 6781.326405] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6781.334663] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6781.378239] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 6782.432333] Lustre: 485478:0:(client.c:1628:after_reply()) @@@ resending request on EINPROGRESS req@00000000b2c38c43 x1842015145851904/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1756682608 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 6782.673420] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6791.342363] LustreError: 485478:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 6791.345079] LustreError: 485478:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 6791.347477] LustreError: 485478:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 6793.704674] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6793.713309] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 6797.436776] LustreError: 486601:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 6797.465523] Lustre: server umount fs15246-MDT0000 complete [ 6798.822096] LustreError: 483240:0:(ldlm_lib.c:1132:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6798.825965] LustreError: 483240:0:(ldlm_lib.c:1132:target_handle_connect()) Skipped 2 previous similar messages [ 6801.257145] LustreError: 487104:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 6801.258840] LustreError: 487104:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6801.273189] Lustre: server umount fs15246-OST0000 complete [ 6809.062343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6809.070250] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6809.070750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6809.077136] Lustre: Skipped 1 previous similar message [ 6811.372150] LustreError: 487604:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6811.373876] LustreError: 487604:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6811.445173] Lustre: server umount lustre-OST0000 complete [ 6813.664516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6813.668886] LustreError: Skipped 1 previous similar message [ 6813.671462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6813.678300] Lustre: Skipped 1 previous similar message [ 6813.681488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6819.297444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6819.300918] Lustre: Skipped 3 previous similar messages [ 6824.417608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6824.421107] Lustre: Skipped 1 previous similar message [ 6826.975115] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6827.007067] LustreError: 487806:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6827.008655] LustreError: 487806:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6827.040875] Lustre: server umount lustre-MDT0000 complete [ 6828.214640] LustreError: 481061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756682637 with bad export cookie 3165323060180281939 [ 6828.217707] LustreError: 481061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6828.217945] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6828.377415] Lustre: server umount lustre-MDT0001 complete [ 6832.550963] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 6833.605992] Key type lgssc unregistered [ 6833.770537] LNet: 488540:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6833.775294] LNetError: 488540:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6833.784859] LNet: Removed LNI 192.168.201.118@tcp [ 6834.302781] Key type .llcrypt unregistered [ 6834.304916] Key type ._llcrypt unregistered [ 6842.059318] Key type ._llcrypt registered [ 6842.060722] Key type .llcrypt registered [ 6842.099757] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6842.498903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6842.506830] alg: No test for adler32 (adler32-zlib) [ 6843.370626] Lustre: Lustre: Build Version: 2.16.58_1_g28cf6eb [ 6843.454176] LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] [ 6845.039172] Key type lgssc registered [ 6845.681242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6848.487902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6850.697388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6852.873894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6854.527500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6861.776826] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 6865.259079] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6870.122737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6873.409469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6876.548402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6879.745765] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6883.036729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6883.070032] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6883.080415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6884.182529] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6884.198962] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6884.245006] Lustre: lustre-MDT0000: new disk, initializing [ 6884.278524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6884.288867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6886.013729] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6891.561355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6891.602936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6891.658329] Lustre: 492943:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6891.697873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6891.701488] Lustre: Skipped 1 previous similar message [ 6891.753316] Lustre: lustre-MDT0001: new disk, initializing [ 6891.801991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6891.820358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6891.825521] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6893.564326] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6897.441098] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6899.106545] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6901.279693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6901.304458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6901.394501] Lustre: lustre-OST0000: new disk, initializing [ 6901.396291] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6901.424269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6902.541868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6902.548287] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6902.600098] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6903.683650] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all [ 6907.309660] Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6908.503769] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6908.569103] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6909.816601] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6909.878606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6912.993749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6912.995139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6912.995253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6912.997308] Lustre: Skipped 1 previous similar message [ 6913.002159] LustreError: Skipped 1 previous similar message [ 6916.589862] LustreError: 495879:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6916.674039] Lustre: server umount lustre-OST0000 complete [ 6918.112501] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6918.112945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6918.115709] Lustre: Skipped 1 previous similar message [ 6918.117297] Lustre: Skipped 1 previous similar message [ 6923.232114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6923.234531] Lustre: Skipped 2 previous similar messages [ 6924.044715] LustreError: 496080:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6924.048937] LustreError: 496080:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6924.085986] Lustre: server umount lustre-MDT0000 complete [ 6925.353675] LustreError: 492936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1756682735 with bad export cookie 9966217649480503969 [ 6925.357334] LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6925.359215] LustreError: 492936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6925.440153] LustreError: 496282:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6925.442213] LustreError: 496282:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6925.534312] Lustre: server umount lustre-MDT0001 complete [ 6936.596234] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_hostid [ 6941.367832] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local [ 6946.719339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6950.553588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6953.754790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6957.310527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6958.219906] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6827 sec ========== 19:26:06 (1756682766) [ 6959.084156] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 19:26:07 (1756682767) === [ 6960.747945] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 19:26:09 (1756682769) === [ 6979.367817] Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local [ 6980.996382] Key type lgssc unregistered [ 6981.223816] LNet: 501133:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6981.228420] LNetError: 501133:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6981.237728] LNet: Removed LNI 192.168.201.118@tcp [ 6981.764792] Key type .llcrypt unregistered [ 6981.766713] Key type ._llcrypt unregistered