[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 618571490 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002062] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004020] kvm-guest: setup PV IPIs [ 0.007291] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008036] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009024] pid_max: default: 32768 minimum: 301 [ 0.010155] LSM: Security Framework initializing [ 0.011071] Yama: becoming mindful. [ 0.012000] SELinux: Initializing. [ 0.012097] *** VALIDATE selinux *** [ 0.021199] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026512] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028150] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029139] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030176] *** VALIDATE tmpfs *** [ 0.031667] *** VALIDATE proc *** [ 0.032317] *** VALIDATE cgroup *** [ 0.033010] *** VALIDATE cgroup2 *** [ 0.034278] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035216] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036050] Spectre V2 : User space: Vulnerable [ 0.037019] Speculative Store Bypass: Vulnerable [ 0.041149] debug: unmapping init [mem 0xffffffff98859000-0xffffffff98860fff] [ 0.044167] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045772] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046029] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050016] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052011] ... event mask: 000000070000000f [ 0.053377] rcu: Hierarchical SRCU implementation. [ 0.055661] smp: Bringing up secondary CPUs ... [ 0.056654] x86: Booting SMP configuration: [ 0.057034] .... node #0, CPUs: #1 #2 #3 [ 0.101860] smp: Brought up 1 node, 4 CPUs [ 0.103019] smpboot: Max logical packages: 1 [ 0.104015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.131124] node 0 deferred pages initialised in 21ms [ 0.136010] devtmpfs: initialized [ 0.137235] x86/mm: Memory block size: 128MB [ 0.140065] gcov: version magic: 0x41383552 [ 0.142318] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.143097] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.144299] pinctrl core: initialized pinctrl subsystem [ 0.145234] [ 0.145980] ************************************************************* [ 0.146019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.147018] ** ** [ 0.148011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.149015] ** ** [ 0.150016] ** This means that this kernel is built to expose internal ** [ 0.151020] ** IOMMU data structures, which may compromise security on ** [ 0.152016] ** your system. ** [ 0.153019] ** ** [ 0.154015] ** If you see this message and you are not debugging the ** [ 0.155018] ** kernel, report this immediately to your vendor! ** [ 0.156012] ** ** [ 0.157020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158014] ************************************************************* [ 0.159754] NET: Registered protocol family 16 [ 0.160689] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.161070] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.162081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.164515] cpuidle: using governor menu [ 0.171898] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.176572] PCI: Using configuration type 1 for base access [ 0.180145] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.194064] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.195023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.197313] cryptd: max_cpu_qlen set to 1000 [ 0.202338] ACPI: Added _OSI(Module Device) [ 0.204018] ACPI: Added _OSI(Processor Device) [ 0.206013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.208017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.213658] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.221683] ACPI: Interpreter enabled [ 0.222075] ACPI: PM: (supports S0 S3 S4 S5) [ 0.223013] ACPI: Using IOAPIC for interrupt routing [ 0.224125] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.225454] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.235885] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.236051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.237025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.238093] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.240428] acpiphp: Slot [2] registered [ 0.241227] acpiphp: Slot [3] registered [ 0.242102] acpiphp: Slot [4] registered [ 0.243120] acpiphp: Slot [5] registered [ 0.244194] acpiphp: Slot [6] registered [ 0.245186] acpiphp: Slot [7] registered [ 0.246163] acpiphp: Slot [8] registered [ 0.247197] acpiphp: Slot [9] registered [ 0.248194] acpiphp: Slot [10] registered [ 0.249181] acpiphp: Slot [11] registered [ 0.250123] acpiphp: Slot [12] registered [ 0.251129] acpiphp: Slot [13] registered [ 0.252124] acpiphp: Slot [14] registered [ 0.253146] acpiphp: Slot [15] registered [ 0.254132] acpiphp: Slot [16] registered [ 0.255335] acpiphp: Slot [17] registered [ 0.256135] acpiphp: Slot [18] registered [ 0.257146] acpiphp: Slot [19] registered [ 0.258079] acpiphp: Slot [20] registered [ 0.259113] acpiphp: Slot [21] registered [ 0.260084] acpiphp: Slot [22] registered [ 0.261095] acpiphp: Slot [23] registered [ 0.262083] acpiphp: Slot [24] registered [ 0.263094] acpiphp: Slot [25] registered [ 0.264081] acpiphp: Slot [26] registered [ 0.265185] acpiphp: Slot [27] registered [ 0.266111] acpiphp: Slot [28] registered [ 0.267091] acpiphp: Slot [29] registered [ 0.268073] acpiphp: Slot [30] registered [ 0.269086] acpiphp: Slot [31] registered [ 0.270057] PCI host bridge to bus 0000:00 [ 0.271021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.272029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.273020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.274021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.275022] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.276024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.277185] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.280500] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.283436] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.290015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.293052] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.294023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.295020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.296018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.297462] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.298666] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.299052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.300733] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.302760] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.308017] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.311017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.314616] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.317022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.320018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.327024] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.333015] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.336018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.339017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.346027] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.352138] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.355019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.358017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.365019] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.372324] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.375019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.378020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.386019] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.392628] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.395018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.398020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.405024] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.416959] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.431021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.443020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.461024] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.474694] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.477849] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.481457] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.483447] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.485285] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.490157] iommu: Default domain type: Passthrough [ 0.493571] SCSI subsystem initialized [ 0.494148] ACPI: bus type USB registered [ 0.495259] usbcore: registered new interface driver usbfs [ 0.498106] usbcore: registered new interface driver hub [ 0.500146] usbcore: registered new device driver usb [ 0.502214] pps_core: LinuxPPS API ver. 1 registered [ 0.505013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.509095] PTP clock support registered [ 0.511091] EDAC MC: Ver: 3.0.0 [ 0.513127] PCI: Using ACPI for IRQ routing [ 0.516082] NetLabel: Initializing [ 0.518010] NetLabel: domain hash size = 128 [ 0.519010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.521087] NetLabel: unlabeled traffic allowed by default [ 0.524109] vgaarb: loaded [ 0.526000] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.526000] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.535213] clocksource: Switched to clocksource kvm-clock [ 0.680413] VFS: Disk quotas dquot_6.6.0 [ 0.682125] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.684821] *** VALIDATE ramfs *** [ 0.686289] *** VALIDATE hugetlbfs *** [ 0.689263] pnp: PnP ACPI init [ 0.691788] pnp: PnP ACPI: found 6 devices [ 0.710236] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.713545] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.715953] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.718211] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.720727] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.722982] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.725793] NET: Registered protocol family 2 [ 0.728357] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.733501] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.737028] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.742435] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.748487] TCP: Hash tables configured (established 65536 bind 65536) [ 0.751703] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.755709] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.760357] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.767688] NET: Registered protocol family 1 [ 0.772143] RPC: Registered named UNIX socket transport module. [ 0.773783] RPC: Registered udp transport module. [ 0.775162] RPC: Registered tcp transport module. [ 0.776648] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.779432] NET: Registered protocol family 44 [ 0.783293] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.785674] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.787857] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.790501] PCI: CLS 0 bytes, default 64 [ 0.792653] Unpacking initramfs... [ 2.500763] debug: unmapping init [mem 0xffff951b3cc54000-0xffff951b3ffbffff] [ 2.506449] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.508782] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.512585] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.053338] Initialise system trusted keyrings [ 3.054985] Key type blacklist registered [ 3.057810] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.066248] zbud: loaded [ 3.070046] *** VALIDATE nfs *** [ 3.071436] *** VALIDATE nfs4 *** [ 3.072974] pstore: using deflate compression [ 3.076388] Platform Keyring initialized [ 3.294728] NET: Registered protocol family 38 [ 3.296394] Key type asymmetric registered [ 3.298061] Asymmetric key parser 'x509' registered [ 3.299986] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.311978] io scheduler mq-deadline registered [ 3.313645] io scheduler kyber registered [ 3.315996] io scheduler bfq registered [ 3.317736] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.330375] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.341453] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.344393] ACPI: Power Button [PWRF] [ 3.458653] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.589147] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.795658] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.922227] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.187197] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.232122] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.279455] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.288565] Non-volatile memory driver v1.3 [ 4.291036] Linux agpgart interface v0.103 [ 4.338470] virtio_blk virtio1: [vda] 133840 512-byte logical blocks (68.5 MB/65.4 MiB) [ 4.342605] vda: detected capacity change from 0 to 68526080 [ 4.364710] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.370205] vdb: detected capacity change from 0 to 1073741824 [ 4.392927] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.397770] vdc: detected capacity change from 0 to 2621440000 [ 4.419737] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.425076] vdd: detected capacity change from 0 to 2621440000 [ 4.448282] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.454102] vde: detected capacity change from 0 to 4294967296 [ 4.472174] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.475310] vdf: detected capacity change from 0 to 4294967296 [ 4.484052] libphy: Fixed MDIO Bus: probed [ 4.491448] usbcore: registered new interface driver usbserial_generic [ 4.494627] usbserial: USB Serial support registered for generic [ 4.496962] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.501995] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.504111] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.506795] mousedev: PS/2 mouse device common for all mice [ 4.510073] rtc_cmos 00:05: RTC can wake from S4 [ 4.513060] rtc_cmos 00:05: registered as rtc0 [ 4.514875] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.519110] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.520670] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.524727] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.533721] intel_pstate: CPU model not supported [ 4.536472] hid: raw HID events driver (C) Jiri Kosina [ 4.538580] usbcore: registered new interface driver usbhid [ 4.540771] usbhid: USB HID core driver [ 4.542471] drop_monitor: Initializing network drop monitor service [ 4.545175] Initializing XFRM netlink socket [ 4.546986] NET: Registered protocol family 10 [ 4.550659] Segment Routing with IPv6 [ 4.552183] NET: Registered protocol family 17 [ 4.555528] mpls_gso: MPLS GSO support [ 4.559842] RAS: Correctable Errors collector initialized. [ 4.562104] AVX version of gcm_enc/dec engaged. [ 4.563740] AES CTR mode by8 optimization enabled [ 4.710079] sched_clock: Marking stable (4710049195, 0)->(6168523234, -1458474039) [ 4.715250] registered taskstats version 1 [ 4.718176] Loading compiled-in X.509 certificates [ 4.721817] zswap: loaded using pool lzo/zbud [ 4.768136] Key type big_key registered [ 4.785781] Key type encrypted registered [ 4.787465] ima: No TPM chip found, activating TPM-bypass! [ 4.789869] ima: Allocated hash algorithm: sha1 [ 4.791767] ima: No architecture policies found [ 4.793763] evm: Initialising EVM extended attributes: [ 4.795844] evm: security.selinux [ 4.797313] evm: security.ima [ 4.798459] evm: security.capability [ 4.799897] evm: HMAC attrs: 0x1 [ 4.803775] rtc_cmos 00:05: setting system clock to 2025-11-17 17:46:34 UTC (1763401594) [ 4.812846] debug: unmapping init [mem 0xffffffff99803000-0xffffffff999fffff] [ 4.828350] debug: unmapping init [mem 0xffffffff98582000-0xffffffff98858fff] [ 4.841179] Write protecting the kernel read-only data: 28672k [ 4.845881] debug: unmapping init [mem 0xffffffff96c03000-0xffffffff96dfffff] [ 4.850590] debug: unmapping init [mem 0xffffffff97514000-0xffffffff975fffff] [ 4.884914] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.895791] systemd[1]: Detected virtualization kvm. [ 4.899727] systemd[1]: Detected architecture x86-64. [ 4.903401] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.936863] systemd[1]: No hostname configured. [ 4.940703] systemd[1]: Set hostname to . [ 4.942817] random: systemd: uninitialized urandom read (16 bytes read) [ 4.945357] systemd[1]: Initializing machine ID from random generator. [ 5.029774] random: ln: uninitialized urandom read (6 bytes read) [ 5.266170] random: systemd: uninitialized urandom read (16 bytes read) [ 5.269651] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 5.274582] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 5.278963] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.766641] device-mapper: uevent: version 1.0.3 [ 6.769480] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.149402] virtio_net virtio0 ens2: renamed from eth0 [ 8.345625] random: fast init done [ 8.612030] scsi host0: ata_piix [ 8.650355] scsi host1: ata_piix [ 8.651790] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.664564] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.334516] random: crng init done [ 14.336087] random: 7 urandom warning(s) missed due to ratelimiting [ 17.322417] dracut-initqueue[595]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 18.503766] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.549653] printk: systemd: 26 output lines suppressed due to ratelimiting [ 20.934264] SELinux: Disabled at runtime. [ 20.992412] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.005130] systemd[1]: Detected virtualization kvm. [ 21.007277] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.979539] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.984674] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.994460] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.005732] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.014515] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.032284] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.055443] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 22.269821] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Reached target Paths. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 23.715761] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 24.883726] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.910373] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.476156] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.673982] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 30.442824] Key type dns_resolver registered [ 30.873365] NFS: Registering the id_resolver key type [ 30.876537] Key type id_resolver registered [ 30.880696] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg330-server login: [ 84.058344] hrtimer: interrupt took 3770887 ns [ 112.428606] libcfs: loading out-of-tree module taints kernel. [ 112.445101] Key type ._llcrypt registered [ 112.446406] Key type .llcrypt registered [ 112.524422] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 127.321506] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 128.445601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 128.457735] alg: No test for adler32 (adler32-zlib) [ 129.652389] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 130.483788] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 132.215164] Key type lgssc registered [ 133.664910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 148.377617] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 149.239169] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 155.286075] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 161.036438] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 168.169425] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 182.154590] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 192.871357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 192.930634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 192.941358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 194.224947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 194.282110] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 194.364845] Lustre: lustre-MDT0000: new disk, initializing [ 194.449840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 194.495646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 197.969338] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.013339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 211.065433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 211.114467] Lustre: 6521:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 211.148209] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 211.151227] Lustre: Skipped 1 previous similar message [ 211.216213] Lustre: lustre-MDT0001: new disk, initializing [ 211.278078] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 211.325382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 211.338848] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 214.668664] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.211821] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 227.519398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 227.574767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 227.728830] Lustre: lustre-OST0000: new disk, initializing [ 227.735878] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 227.805758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 229.644823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 229.655352] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 229.738340] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 232.694570] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.004869] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 245.084680] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 245.152510] Lustre: lustre-OST0001: new disk, initializing [ 245.155281] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 245.228638] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 250.008588] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.906370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 250.915506] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 251.007252] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 260.874507] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 265.276938] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 276.046397] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing check_logdir /tmp/testlogs/ [ 280.275225] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing yml_node [ 284.973219] Lustre: DEBUG MARKER: Client: 2.16.61.42 [ 287.295996] Lustre: DEBUG MARKER: MDS: 2.16.61.42 [ 289.731955] Lustre: DEBUG MARKER: OSS: 2.16.61.42 [ 291.189225] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Nov 17 12:51:20 EST 2025 [ 306.621657] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 307.966306] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 324.064133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 324.079117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 324.095088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 327.648082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 327.651894] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 327.666373] Lustre: Skipped 1 previous similar message [ 332.782392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 332.785159] Lustre: Skipped 3 previous similar messages [ 337.375408] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 337.503529] LustreError: 12225:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 337.596071] Lustre: server umount lustre-MDT0000 complete [ 340.986805] LustreError: 7903:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763401930 with bad export cookie 12149785093489242936 [ 340.987217] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 341.002303] LustreError: 7903:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 343.008450] LustreError: 6527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 343.012295] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 343.012734] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 343.045114] LustreError: 6527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 343.046647] Lustre: Skipped 2 previous similar messages [ 345.060437] LustreError: 6527:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 345.077258] LustreError: 6527:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 347.235317] LustreError: 12429:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 347.261500] LustreError: 12429:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 347.441527] Lustre: server umount lustre-MDT0001 complete [ 351.365210] LustreError: 12632:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 351.367635] LustreError: 12632:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 351.431121] Lustre: server umount lustre-OST0000 complete [ 354.892504] LustreError: 12832:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 354.897571] LustreError: 12832:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 355.075588] Lustre: server umount lustre-OST0001 complete [ 360.450977] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 368.392849] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 376.750620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 382.992740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 389.320493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 394.876896] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 401.579719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 401.642103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 401.940218] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 402.003785] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 402.168721] Lustre: lustre-MDT0000: new disk, initializing [ 402.316208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 402.344323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 406.050532] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 416.844146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 416.926899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 417.025776] Lustre: 15014:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 417.114232] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 417.124502] Lustre: Skipped 1 previous similar message [ 417.181734] Lustre: lustre-MDT0001: new disk, initializing [ 417.288944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 417.341462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 417.358435] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 421.305965] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 428.601074] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 431.847404] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 438.171456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 438.264961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 438.492114] Lustre: lustre-OST0000: new disk, initializing [ 438.513119] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 438.634169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 439.850280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 439.863448] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 439.971406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 444.594246] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 451.086860] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 454.689108] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 454.856749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 457.947563] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 458.111859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 460.768297] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 460.772898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 460.779644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 460.795294] Lustre: Skipped 1 previous similar message [ 460.799180] Lustre: Skipped 3 previous similar messages [ 466.124727] LustreError: 17954:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 466.126871] LustreError: 17954:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 466.258592] Lustre: server umount lustre-OST0000 complete [ 471.008180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 471.026142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 471.048834] Lustre: Skipped 4 previous similar messages [ 476.285151] LustreError: 18156:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 476.287479] LustreError: 18156:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 476.384190] Lustre: server umount lustre-MDT0000 complete [ 479.900985] LustreError: 15008:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763402069 with bad export cookie 12149785093489244469 [ 479.904935] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 479.913174] LustreError: 15008:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 480.278642] Lustre: server umount lustre-MDT0001 complete [ 483.875633] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 12:54:33 (1763402073) [ 485.916323] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 487.386994] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 12:54:36 (1763402076) [ 493.252814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 493.832400] LustreError: 19027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 493.907605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 497.391102] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 499.169159] LustreError: 19027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.295462] LustreError: 19027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.133224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 510.178314] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 515.075688] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 518.496766] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 525.220350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 531.032874] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 536.230874] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 549.344132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 549.350314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 549.362134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 549.375410] Lustre: Skipped 2 previous similar messages [ 549.401356] Lustre: Skipped 3 previous similar messages [ 553.741903] LustreError: 21274:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 553.763441] LustreError: 21274:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 553.946941] Lustre: server umount lustre-OST0000 complete [ 558.049098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 563.796719] Lustre: server umount lustre-MDT0000 complete [ 564.705561] LustreError: 20236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 564.724831] LustreError: 20236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 567.448823] LustreError: 19007:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763402157 with bad export cookie 12149785093489245351 [ 567.458125] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 567.473755] LustreError: 19007:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 567.836370] Lustre: server umount lustre-MDT0001 complete [ 578.154831] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 580.911550] Key type lgssc unregistered [ 581.200203] LNet: 22211:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 581.207527] LNetError: 22211:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 581.242256] LNet: Removed LNI 192.168.203.130@tcp [ 582.211513] Key type .llcrypt unregistered [ 582.213581] Key type ._llcrypt unregistered [ 600.567852] Key type ._llcrypt registered [ 600.569833] Key type .llcrypt registered [ 600.658027] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 601.928928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 601.959035] alg: No test for adler32 (adler32-zlib) [ 602.985677] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 603.161826] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 604.799193] Key type lgssc registered [ 605.742294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 613.332182] loop: module loaded [ 613.416075] loop0: detected capacity change from 0 to 204800000 [ 613.627538] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 613.660520] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 613.872518] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 613.921807] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 619.118585] loop0: detected capacity change from 0 to 204800000 [ 619.391585] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 619.413504] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 619.441126] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 619.597552] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 624.359663] loop0: detected capacity change from 0 to 204800000 [ 629.882039] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 629.972224] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 629.986547] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 631.150442] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 631.188720] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 631.240576] Lustre: lustre-MDT0000: new disk, initializing [ 631.320815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 631.338379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 634.846354] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 643.054132] loop1: detected capacity change from 0 to 204800000 [ 648.085253] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 648.143851] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 648.391642] Lustre: lustre-OST07c6: new disk, initializing [ 648.402913] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 648.406345] Lustre: Skipped 1 previous similar message [ 648.496256] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 653.820247] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 657.479803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 657.495968] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 657.536917] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 663.474331] Lustre: 24354:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 663.478591] Lustre: 24354:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 663.482051] Lustre: 24354:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 663.512464] Lustre: 24354:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 663.516584] Lustre: 24354:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 663.531283] Lustre: 24354:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 680.840529] Lustre: 24352:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 680.850553] Lustre: 24352:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 680.857384] Lustre: 24352:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 680.868525] Lustre: 24352:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 680.884077] Lustre: 24352:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 680.895139] Lustre: 24352:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 685.503833] Lustre: 24352:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 685.508327] Lustre: 24352:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 685.513028] Lustre: 24352:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 685.516963] Lustre: 24352:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.521253] Lustre: 24352:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 685.526745] Lustre: 24352:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.535231] Lustre: 24352:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 685.540306] Lustre: 24352:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.546471] Lustre: 24352:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 685.553349] Lustre: 24352:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.565565] Lustre: 24352:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 685.579877] Lustre: 24352:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 688.809644] Lustre: 24352:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 688.828880] Lustre: 24352:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 688.839313] Lustre: 24352:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 688.855639] Lustre: 24352:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 688.865714] Lustre: 24352:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 688.875159] Lustre: 24352:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 688.887358] Lustre: 24352:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 688.899219] Lustre: 24352:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 688.911333] Lustre: 24352:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 688.929834] Lustre: 24352:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 688.953223] Lustre: 24352:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 688.974638] Lustre: 24352:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.475980] Lustre: 25944:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 693.482272] Lustre: 25944:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 693.487348] Lustre: 25944:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 693.491840] Lustre: 25944:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.495512] Lustre: 25944:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 693.500216] Lustre: 25944:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.504714] Lustre: 25944:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 693.508570] Lustre: 25944:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.513452] Lustre: 25944:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 693.522206] Lustre: 25944:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.527326] Lustre: 25944:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 693.538182] Lustre: 25944:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 704.335214] Lustre: 25944:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 704.359192] Lustre: 25944:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 5 previous similar messages [ 704.373968] Lustre: 25944:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 704.397322] Lustre: 25944:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 704.421424] Lustre: 25944:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 704.441124] Lustre: 25944:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 704.454796] Lustre: 25944:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 704.468442] Lustre: 25944:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 704.478517] Lustre: 25944:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 704.485480] Lustre: 25944:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 704.497330] Lustre: 25944:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 704.509771] Lustre: 25944:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 754.656390] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 754.663423] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 754.679335] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 757.821754] LustreError: 26540:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 758.000221] Lustre: server umount lustre-OST07c6 complete [ 767.756393] LustreError: 27042:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 767.759658] LustreError: 27042:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 767.967654] Lustre: server umount lustre-MDT0000 complete [ 783.032949] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 785.377304] Key type lgssc unregistered [ 785.650538] LNet: 27875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.653686] LNetError: 27875:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 785.678836] LNet: Removed LNI 192.168.203.130@tcp [ 786.359290] Key type .llcrypt unregistered [ 786.362467] Key type ._llcrypt unregistered [ 800.578540] Key type ._llcrypt registered [ 800.580249] Key type .llcrypt registered [ 800.690214] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 801.573685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 801.625879] alg: No test for adler32 (adler32-zlib) [ 802.609385] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 802.865949] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 804.519174] Key type lgssc registered [ 805.407801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 811.162626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 811.177151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 812.587483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 816.158909] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 822.755111] LustreError: 29105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 822.769441] LustreError: 29105:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 823.267187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 823.588787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 826.329266] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 830.724284] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 833.480483] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 838.584523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 838.819708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 843.546056] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 848.232521] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 858.283952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 869.868276] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 869.880703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 869.899323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 873.953965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 873.954302] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 873.957519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 875.583269] LustreError: 31573:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 875.668488] Lustre: server umount lustre-OST0000 complete [ 879.072384] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 879.081368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 879.082837] Lustre: Skipped 1 previous similar message [ 879.086248] Lustre: Skipped 1 previous similar message [ 884.192276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 884.198023] Lustre: Skipped 1 previous similar message [ 884.753479] LustreError: 31787:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 884.769877] LustreError: 31787:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 884.898798] Lustre: server umount lustre-MDT0000 complete [ 888.216110] LustreError: 29084:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763402477 with bad export cookie 16025175572666151219 [ 888.219743] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 888.224904] LustreError: 29084:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 888.388823] LustreError: 31988:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 888.396926] LustreError: 31988:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 888.608216] Lustre: server umount lustre-MDT0001 complete [ 898.330236] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 900.931567] Key type lgssc unregistered [ 901.193197] LNet: 32525:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.209285] LNetError: 32525:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.229752] LNet: Removed LNI 192.168.203.130@tcp [ 902.129070] Key type .llcrypt unregistered [ 902.131372] Key type ._llcrypt unregistered [ 918.800626] Key type ._llcrypt registered [ 918.802464] Key type .llcrypt registered [ 918.873498] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 920.501326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 920.566452] alg: No test for adler32 (adler32-zlib) [ 921.621259] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 921.808653] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 923.463167] Key type lgssc registered [ 924.558372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 930.923848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 930.942174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 932.358856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 935.943681] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 942.568572] LustreError: 33782:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 942.587678] LustreError: 33782:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 943.629033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 944.070552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 947.274948] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 951.566424] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 954.628299] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 961.201051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 961.609808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 966.169436] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 971.466742] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 978.095481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 993.765146] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 993.769070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 993.776330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 993.788387] Lustre: Skipped 1 previous similar message [ 996.031341] LustreError: 36054:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 996.156950] Lustre: server umount lustre-OST0000 complete [ 1000.419400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1000.423552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1000.440942] Lustre: Skipped 1 previous similar message [ 1000.453262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1004.006249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1004.007279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1005.511910] LustreError: 36255:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1005.519845] LustreError: 36255:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1005.641295] Lustre: server umount lustre-MDT0000 complete [ 1007.014604] LustreError: 35156:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1007.038864] LustreError: 35156:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1009.002268] LustreError: 33761:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763402598 with bad export cookie 10833243558918143305 [ 1009.021586] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1009.170153] LustreError: 36456:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1009.172721] LustreError: 36456:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1009.348462] Lustre: server umount lustre-MDT0001 complete [ 1018.880585] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 1021.335252] Key type lgssc unregistered [ 1021.617842] LNet: 36988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1021.638125] LNetError: 36988:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1021.655466] LNet: Removed LNI 192.168.203.130@tcp [ 1022.525098] Key type .llcrypt unregistered [ 1022.528671] Key type ._llcrypt unregistered [ 1039.141034] Key type ._llcrypt registered [ 1039.146466] Key type .llcrypt registered [ 1039.245783] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1040.065944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1040.115405] alg: No test for adler32 (adler32-zlib) [ 1041.086457] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 1041.273092] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 1042.911170] Key type lgssc registered [ 1043.711179] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1049.752203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1049.764701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1051.269644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1053.915271] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1060.212359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1060.561708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1063.813339] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1067.873053] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1070.656195] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1076.543096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1076.870421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1077.867781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1081.265307] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1085.639761] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1091.555882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1091.563464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1091.563914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1091.571470] Lustre: Skipped 1 previous similar message [ 1093.288947] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.30@tcp (stopping) [ 1093.291170] Lustre: Skipped 2 previous similar messages [ 1093.674241] LustreError: 40608:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1093.780450] Lustre: server umount lustre-MDT0000 complete [ 1096.672329] LustreError: 38363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1096.684883] LustreError: 38363:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1096.828620] LustreError: 38343:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763402686 with bad export cookie 7047313921524078536 [ 1096.848866] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1098.380922] LustreError: 38800:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1098.391477] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.30@tcp (stopping) [ 1101.792399] LustreError: 38360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1101.793629] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1101.829530] Lustre: Skipped 1 previous similar message [ 1101.836721] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1103.025509] LustreError: 40810:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1103.028216] LustreError: 40810:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1103.144303] Lustre: server umount lustre-MDT0001 complete [ 1122.990905] LustreError: 41032:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1123.002260] LustreError: 41032:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1123.188426] Lustre: server umount lustre-OST0000 complete [ 1136.118410] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 1138.648447] Key type lgssc unregistered [ 1138.877900] LNet: 41870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1138.887626] LNetError: 41870:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1138.897912] LNet: Removed LNI 192.168.203.130@tcp [ 1139.453397] Key type .llcrypt unregistered [ 1139.456170] Key type ._llcrypt unregistered [ 1156.924097] Key type ._llcrypt registered [ 1156.925859] Key type .llcrypt registered [ 1157.071902] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1157.934801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1157.986556] alg: No test for adler32 (adler32-zlib) [ 1159.008206] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 1159.258368] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 1160.921285] Key type lgssc registered [ 1161.879328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1168.174808] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1168.206158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1169.649845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1172.815840] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1179.084408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1179.320543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1182.104742] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1186.093334] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1188.656811] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1193.998871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1194.254487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1196.350729] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1198.708910] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1202.970523] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1208.804225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1208.807755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1208.809726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1208.811825] Lustre: Skipped 1 previous similar message [ 1209.500302] Lustre: lustre-OST0000: Not available for connect from 192.168.203.30@tcp (stopping) [ 1209.507573] Lustre: Skipped 1 previous similar message [ 1213.750092] LustreError: 45491:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1213.822247] Lustre: server umount lustre-OST0000 complete [ 1230.817267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1230.831327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1230.861024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1234.400196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1234.406813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1236.671270] LustreError: 45868:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1236.676789] LustreError: 45868:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1236.854987] Lustre: server umount lustre-MDT0000 complete [ 1239.525337] LustreError: 43247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1239.548528] LustreError: 43247:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1240.404233] LustreError: 43227:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763402830 with bad export cookie 11165349287022508521 [ 1240.426959] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1240.441209] LustreError: 43227:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1246.662529] LustreError: 46070:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1246.665131] LustreError: 46070:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1246.955886] Lustre: server umount lustre-MDT0001 complete [ 1255.455629] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 1257.811442] Key type lgssc unregistered [ 1258.076905] LNet: 46601:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1258.084951] LNetError: 46601:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1258.102688] LNet: Removed LNI 192.168.203.130@tcp [ 1258.816594] Key type .llcrypt unregistered [ 1258.819855] Key type ._llcrypt unregistered [ 1277.634158] Key type ._llcrypt registered [ 1277.635814] Key type .llcrypt registered [ 1277.717393] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1278.455217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1278.465607] alg: No test for adler32 (adler32-zlib) [ 1279.472887] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 1279.659479] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 1281.287896] Key type lgssc registered [ 1282.389891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1288.156423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1288.172287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1289.477221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1292.049371] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1297.037828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1297.272878] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1299.409309] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1303.256448] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1305.492049] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1310.443682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1310.677250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1314.726504] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1315.828732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1319.043864] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1326.963945] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1329.501127] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1341.026653] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-11-1713h08m49s [ 1342.185978] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1346.205657] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.30@tcp (stopping) [ 1347.273340] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.30@tcp (stopping) [ 1347.554309] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1347.564345] Lustre: Skipped 1 previous similar message [ 1348.644447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1348.647624] Lustre: Skipped 5 previous similar messages [ 1349.795521] LustreError: 50541:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1349.861381] Lustre: server umount lustre-MDT0000 complete [ 1352.332595] LustreError: 47972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1352.349076] LustreError: 47972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1352.418714] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1354.733253] LustreError: 48999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1354.743137] LustreError: 48999:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1357.275367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1357.388576] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1357.542824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1357.611439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1358.566695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1360.345351] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1362.638099] Lustre: DEBUG MARKER: Wait for df (36011) ... [ 1363.681860] Lustre: DEBUG MARKER: done [ 1368.032623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1368.034296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1368.041137] LustreError: Skipped 1 previous similar message [ 1368.043362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1368.063873] Lustre: Skipped 2 previous similar messages [ 1368.072708] Lustre: Skipped 1 previous similar message [ 1374.180712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1374.185232] Lustre: Skipped 3 previous similar messages [ 1380.319172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1380.353469] LustreError: 51591:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1380.363917] LustreError: 51591:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1380.458665] Lustre: server umount lustre-OST0000 complete [ 1384.422357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1384.425349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1384.441171] Lustre: Skipped 1 previous similar message [ 1384.444840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1384.454881] Lustre: Skipped 2 previous similar messages [ 1389.157202] LustreError: 51798:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1389.166723] LustreError: 51798:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1389.257967] Lustre: server umount lustre-MDT0000 complete [ 1390.561202] LustreError: 47978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1390.578125] LustreError: 47978:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1391.641938] LustreError: 47957:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763402981 with bad export cookie 16403708130459380908 [ 1391.643511] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1391.649797] LustreError: 47957:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1391.746691] LustreError: 52000:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1391.751058] LustreError: 52000:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1391.884635] Lustre: server umount lustre-MDT0001 complete [ 1397.372804] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 1398.979154] Key type lgssc unregistered [ 1399.159737] LNet: 52531:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1399.163511] LNetError: 52531:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1399.173068] LNet: Removed LNI 192.168.203.130@tcp [ 1399.653485] Key type .llcrypt unregistered [ 1399.655045] Key type ._llcrypt unregistered [ 1422.244325] Key type ._llcrypt registered [ 1422.245879] Key type .llcrypt registered [ 1422.307416] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 1433.204503] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1433.997972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1434.120810] alg: No test for adler32 (adler32-zlib) [ 1435.179728] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 1435.364021] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 1437.047181] Key type lgssc registered [ 1437.874777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1444.912766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1450.566317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1455.538379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1460.783864] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1466.805691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1466.906942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1466.921835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1468.304619] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1468.365052] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1468.545491] Lustre: lustre-MDT0000: new disk, initializing [ 1468.682498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1468.717577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1472.282719] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1481.094318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1481.189640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1481.280251] Lustre: 56456:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1481.344164] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1481.348272] Lustre: Skipped 1 previous similar message [ 1481.410808] Lustre: lustre-MDT0001: new disk, initializing [ 1481.519341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1481.557212] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1481.566212] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1484.684704] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1490.494613] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1493.483505] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1499.234650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1499.298442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1499.532665] Lustre: lustre-OST0000: new disk, initializing [ 1499.539132] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1499.634861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1504.571669] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1506.371194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1506.377779] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1506.473026] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1510.473776] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1513.565863] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1513.747203] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1516.762497] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1516.944459] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1522.146897] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1522.154989] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1522.180239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1522.182759] Lustre: Skipped 1 previous similar message [ 1524.470479] LustreError: 59394:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1524.544599] Lustre: server umount lustre-OST0000 complete [ 1527.776030] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1527.786503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1527.801981] Lustre: Skipped 1 previous similar message [ 1527.806444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1532.384480] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1532.384684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1532.416963] Lustre: Skipped 1 previous similar message [ 1537.504169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1537.510756] Lustre: Skipped 1 previous similar message [ 1541.607147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1541.610450] Lustre: Skipped 1 previous similar message [ 1542.111269] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1542.215967] LustreError: 59596:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1542.219233] LustreError: 59596:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1542.361102] Lustre: server umount lustre-MDT0000 complete [ 1546.336068] LustreError: 56449:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763403136 with bad export cookie 2005873785152776998 [ 1546.389362] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1546.397251] LustreError: 56449:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1546.543699] LustreError: 59798:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1546.565458] LustreError: 59798:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1546.864120] Lustre: server umount lustre-MDT0001 complete [ 1556.644039] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 13:12:25 (1763403145) [ 1564.191346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1564.818981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1569.066694] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1570.283731] LustreError: 60471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1570.290752] LustreError: 60471:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1574.373433] LustreError: 60471:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1576.365911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1576.641577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1579.959917] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1584.934191] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1588.291258] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1595.670771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1596.142224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1601.578405] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1606.552241] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1616.446504] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-11-1713h13m25s [ 1617.913968] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1623.761970] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1625.844333] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1625.896319] LustreError: 60467:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1646.262733] Lustre: lustre-MDT0000: Client a712ef23-1907-4526-87cc-e7b63f4b14f4 (at 192.168.203.30@tcp) reconnecting [ 1665.975120] LustreError: 60467:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1667.787191] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1669.625275] Lustre: DEBUG MARKER: done [ 1674.208847] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1674.217797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1674.236953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1676.262538] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1679.735714] LustreError: 63476:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1679.740441] LustreError: 63476:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1679.876359] Lustre: server umount lustre-OST0000 complete [ 1684.449089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1684.461756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1684.489133] Lustre: Skipped 1 previous similar message [ 1689.195389] LustreError: 63677:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1689.203226] LustreError: 63677:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1689.293625] Lustre: server umount lustre-MDT0000 complete [ 1692.185940] LustreError: 60450:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763403281 with bad export cookie 2005873785152777880 [ 1692.187948] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1692.197867] LustreError: 60450:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1692.484624] Lustre: server umount lustre-MDT0001 complete [ 1700.973352] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 1703.136823] Key type lgssc unregistered [ 1703.426835] LNet: 64411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.439092] LNetError: 64411:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1703.455528] LNet: Removed LNI 192.168.203.130@tcp [ 1704.069351] Key type .llcrypt unregistered [ 1704.071067] Key type ._llcrypt unregistered [ 1730.675113] Key type ._llcrypt registered [ 1730.676809] Key type .llcrypt registered [ 1730.777214] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 1742.733777] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1743.700865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1743.708418] alg: No test for adler32 (adler32-zlib) [ 1744.792133] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 1744.978536] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 1746.631525] Key type lgssc registered [ 1747.512512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1755.610750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1761.600246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1766.619262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1772.172900] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1778.759689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1778.806038] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1778.831502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1780.148554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1780.193125] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1780.270365] Lustre: lustre-MDT0000: new disk, initializing [ 1780.374350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1780.385479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1783.775422] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1792.665276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1792.772833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1792.866513] Lustre: 68340:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1792.938958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1792.941760] Lustre: Skipped 1 previous similar message [ 1793.000726] Lustre: lustre-MDT0001: new disk, initializing [ 1793.077613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1793.104413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1793.117330] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1796.202356] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1802.562365] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1805.591947] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1811.429561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1811.485401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1811.660460] Lustre: lustre-OST0000: new disk, initializing [ 1811.673612] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1811.735227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1815.716887] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1821.046759] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1821.752598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1821.766215] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1821.876163] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1824.109148] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1824.266838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1826.959097] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1827.118371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1832.417431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1832.417959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1832.421633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1832.436671] Lustre: Skipped 1 previous similar message [ 1834.487344] LustreError: 71279:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1834.598741] Lustre: server umount lustre-OST0000 complete [ 1837.546070] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1837.553653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1837.556029] Lustre: Skipped 1 previous similar message [ 1842.658070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1842.669451] Lustre: Skipped 1 previous similar message [ 1843.630215] LustreError: 71480:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1843.638450] LustreError: 71480:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1843.784863] Lustre: server umount lustre-MDT0000 complete [ 1846.816659] LustreError: 69120:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763403436 with bad export cookie 7996760699559625918 [ 1846.819631] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1846.832819] LustreError: 69120:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1846.988601] LustreError: 71681:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1846.991299] LustreError: 71681:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1847.194683] Lustre: server umount lustre-MDT0001 complete [ 1854.365121] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 13:17:23 (1763403443) [ 1855.660263] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1857.149443] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 13:17:26 (1763403446) [ 1867.821359] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 1871.551245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1871.907175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1872.736086] Lustre: Failing over lustre-MDT0000 [ 1872.746844] LustreError: 72757:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1872.752090] LustreError: 72757:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1872.756532] LustreError: 72757:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 1872.878392] LustreError: 72808:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1872.881569] LustreError: 72808:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1873.061244] Lustre: server umount lustre-MDT0000 complete [ 1878.556886] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 13:17:47 (1763403467) [ 1884.683878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1885.102296] LustreError: 73376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1885.121407] LustreError: 73376:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1885.177602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1888.204308] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1890.276955] LustreError: 73376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1894.369469] LustreError: 73376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1894.383644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1894.592923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1897.357444] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1901.008966] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1903.668222] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1909.290811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1913.873947] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1918.369547] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1923.523930] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1930.212369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1930.225074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1930.227483] Lustre: Skipped 2 previous similar messages [ 1930.799161] LustreError: 75721:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1930.801250] LustreError: 75721:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1930.891791] Lustre: server umount lustre-MDT0000 complete [ 1933.762987] LustreError: 73355:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763403523 with bad export cookie 7996760699559627073 [ 1933.774444] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1934.080753] Lustre: server umount lustre-MDT0001 complete [ 1936.614178] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1939.754463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1947.070627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1951.711165] Lustre: 66553:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763403525/real 1763403525] req@00000000eec6b24b x1849062414416384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763403541 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1951.751482] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1951.777594] Lustre: Skipped 1 previous similar message [ 1954.783174] Lustre: 66551:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763403528/real 1763403528] req@00000000be52107d x1849062414416512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763403544 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1956.895365] Lustre: 66551:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763403530/real 1763403530] req@0000000003207589 x1849062414416640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763403546 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1958.944930] LustreError: 66549:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008e9c0f82 x1849062414417536/t0(0) o250->MGC192.168.203.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1959.202422] LustreError: 74984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1959.289503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1959.295251] Lustre: Skipped 1 previous similar message [ 1959.372555] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1960.287846] Lustre: 66552:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763403533/real 1763403533] req@000000008398331d x1849062414416896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763403549 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1962.250425] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1964.383206] Lustre: 66553:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763403538/real 1763403538] req@000000004eec91e5 x1849062414417024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763403554 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1966.562225] LustreError: 76781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1966.574445] LustreError: 76781:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1969.732698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1971.111838] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1973.130620] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 1977.969959] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1981.089233] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1988.837480] LustreError: 78181:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1988.853218] LustreError: 78181:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1988.956386] Lustre: server umount lustre-MDT0000 complete [ 1990.624297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1990.629983] LustreError: Skipped 1 previous similar message [ 1990.633232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1990.642133] LustreError: 74984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1991.881801] LustreError: 73356:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763403581 with bad export cookie 7996760699559628977 [ 1991.886607] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1991.895502] LustreError: 73356:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 1992.266978] Lustre: server umount lustre-MDT0001 complete [ 1998.550734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2006.765147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2013.151288] Lustre: 66552:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763403586/real 1763403586] req@000000006089905b x1849062414442112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763403602 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2013.179729] Lustre: 66552:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2013.192282] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2013.209789] Lustre: Skipped 2 previous similar messages [ 2017.311710] LustreError: 66549:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f2f2df7d x1849062414443008/t0(0) o250->MGC192.168.203.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2017.581134] LustreError: 77501:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2017.600025] LustreError: 77501:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2017.648265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2017.654429] Lustre: Skipped 1 previous similar message [ 2017.695367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2020.647096] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2027.987866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2031.351262] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2031.354469] Lustre: Skipped 1 previous similar message [ 2031.441744] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2036.415789] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2039.478622] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2046.503667] LustreError: 80638:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2046.507900] LustreError: 80638:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2046.580772] Lustre: server umount lustre-MDT0000 complete [ 2046.947833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2046.956741] Lustre: Skipped 1 previous similar message [ 2049.599552] LustreError: 73355:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763403639 with bad export cookie 7996760699559630349 [ 2049.601726] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2049.622814] LustreError: 73355:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2049.940957] Lustre: server umount lustre-MDT0001 complete [ 2055.773734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2059.020426] Lustre: DEBUG MARKER: files compared the same [ 2070.901810] Lustre: server umount lustre-OST0000 complete [ 2085.571474] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 2088.143654] Key type lgssc unregistered [ 2088.382794] LNet: 82271:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2088.400764] LNetError: 82271:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2088.415112] LNet: Removed LNI 192.168.203.130@tcp [ 2089.116526] Key type .llcrypt unregistered [ 2089.118859] Key type ._llcrypt unregistered [ 2107.693282] Key type ._llcrypt registered [ 2107.694823] Key type .llcrypt registered [ 2107.780472] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2108.481344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2108.511113] alg: No test for adler32 (adler32-zlib) [ 2109.531198] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 2109.726798] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 2111.375179] Key type lgssc registered [ 2112.436337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2119.766734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2119.784347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2121.210829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2124.548204] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug +malloc all [ 2130.404095] LustreError: 83646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2130.434165] LustreError: 83646:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2131.673960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2131.998381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2135.659586] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug +malloc all [ 2140.543151] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2143.816376] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2149.226130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2149.414383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2153.778951] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug +malloc all [ 2158.700983] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2165.804956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2175.970403] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2175.973051] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2175.980369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2175.996809] Lustre: Skipped 1 previous similar message [ 2181.089928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2181.095691] Lustre: Skipped 2 previous similar messages [ 2181.250477] LustreError: 85911:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2181.378268] Lustre: server umount lustre-OST0000 complete [ 2186.209555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2186.213038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2186.230571] Lustre: Skipped 1 previous similar message [ 2191.106178] LustreError: 86113:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2191.150083] LustreError: 86113:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2191.283498] Lustre: server umount lustre-MDT0000 complete [ 2191.330543] LustreError: 84360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2191.342126] LustreError: 84360:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2194.305944] LustreError: 83627:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763403784 with bad export cookie 1789471366490888230 [ 2194.306518] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2194.320698] LustreError: 83627:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2194.477731] LustreError: 86315:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2194.492409] LustreError: 86315:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2194.683598] Lustre: server umount lustre-MDT0001 complete [ 2203.141684] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 2205.237326] Key type lgssc unregistered [ 2205.461401] LNet: 86848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2205.477650] LNetError: 86848:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2205.491083] LNet: Removed LNI 192.168.203.130@tcp [ 2206.120757] Key type .llcrypt unregistered [ 2206.122853] Key type ._llcrypt unregistered [ 2225.546863] Key type ._llcrypt registered [ 2225.549876] Key type .llcrypt registered [ 2225.636667] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2226.217421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2226.276481] alg: No test for adler32 (adler32-zlib) [ 2227.230564] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 2227.430485] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 2229.071493] Key type lgssc registered [ 2229.920662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2236.488602] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2236.507939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2252.063317] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2257.186258] LustreError: 88200:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.130@tcp: failed processing log, type 4: rc = -110 [ 2283.871273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2289.492627] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2294.520647] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2302.450181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2302.605765] Lustre: *** cfs_fail_loc=706, val=0*** [ 2303.044050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2303.171848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2306.371763] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2308.078164] LustreError: 88227:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2308.093988] LustreError: 88227:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2312.715513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2313.008954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2316.303571] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2320.907497] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2323.865530] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2328.546516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2328.549152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2328.563941] LustreError: Skipped 1 previous similar message [ 2328.584520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2329.569760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2329.581630] Lustre: Skipped 1 previous similar message [ 2331.443693] LustreError: 90529:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2331.549292] Lustre: server umount lustre-OST0000 complete [ 2334.691338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2334.701388] Lustre: Skipped 2 previous similar messages [ 2334.703862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2334.712295] Lustre: Skipped 2 previous similar messages [ 2339.810853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2339.821297] Lustre: Skipped 1 previous similar message [ 2340.711415] LustreError: 90730:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2340.716528] LustreError: 90730:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2340.777103] Lustre: server umount lustre-MDT0000 complete [ 2343.254924] LustreError: 88209:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763403932 with bad export cookie 10444825446715994420 [ 2343.258377] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2343.264069] LustreError: 88209:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2343.380798] LustreError: 90931:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2343.392347] LustreError: 90931:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2343.524439] Lustre: server umount lustre-MDT0001 complete [ 2350.882670] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 2352.958739] Key type lgssc unregistered [ 2353.167410] LNet: 91464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2353.175707] LNetError: 91464:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2353.191658] LNet: Removed LNI 192.168.203.130@tcp [ 2353.828132] Key type .llcrypt unregistered [ 2353.829821] Key type ._llcrypt unregistered [ 2372.524929] Key type ._llcrypt registered [ 2372.526828] Key type .llcrypt registered [ 2372.633808] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2373.391859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2373.484789] alg: No test for adler32 (adler32-zlib) [ 2374.569851] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 2374.852928] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 2376.572484] Key type lgssc registered [ 2377.437436] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2383.318558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2383.336300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2387.197763] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2392.145551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2392.501341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2395.228634] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2397.671589] LustreError: 93510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2397.679865] LustreError: 93510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2401.782479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2402.084749] LustreError: 93514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2402.096115] LustreError: 93514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2402.110131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2403.108273] LustreError: 93509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2403.129285] LustreError: 93509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2406.377251] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2408.418581] LustreError: 93509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2408.429091] LustreError: 93509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2412.616722] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2412.627280] Lustre: MGS: Client 005e768b-b990-4cfb-9ac6-db475c58dbd8 (at 0@lo) reconnecting [ 2412.830361] LustreError: 93524:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2412.836826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2412.908139] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2416.385140] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2428.384921] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2428.387376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2428.389426] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2428.406984] Lustre: Skipped 1 previous similar message [ 2433.398124] LustreError: 95473:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2433.470970] Lustre: server umount lustre-OST0000 complete [ 2438.624523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2438.627851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2438.637745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2438.637753] Lustre: Skipped 1 previous similar message [ 2438.640087] Lustre: Skipped 1 previous similar message [ 2443.748111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2443.750948] Lustre: Skipped 1 previous similar message [ 2448.867056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2448.873384] Lustre: Skipped 1 previous similar message [ 2450.399175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2450.461584] LustreError: 95674:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2450.463756] LustreError: 95674:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2450.481263] Lustre: server umount lustre-MDT0000 complete [ 2453.877644] LustreError: 95878:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2453.884199] LustreError: 95878:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2454.000342] Lustre: server umount lustre-MDT0001 complete [ 2456.606467] LustreError: 96078:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2456.609157] LustreError: 96078:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2456.657324] Lustre: server umount MGS complete [ 2464.574475] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 13:27:33 (1763404053) [ 2486.058426] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 2494.590909] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2503.822897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2509.411686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2514.446724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2519.642456] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2525.162900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2525.252605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2529.459936] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2537.338303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2537.395819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2537.547728] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2537.589137] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2537.653190] Lustre: lustre-MDT0001: new disk, initializing [ 2537.753675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2537.773346] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2537.778247] Lustre: 100594:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2537.783620] LustreError: 100594:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2540.936926] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2548.561828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2548.622985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2548.774088] Lustre: lustre-OST0000: new disk, initializing [ 2548.783323] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2548.851869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2554.079205] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2560.033853] Lustre: 101896:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000057279ee8 x1849063074993664/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763404165 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2560.078943] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2566.062746] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2566.082438] Lustre: MGS: Client f956e915-5e1f-47aa-b7f8-b32ae8ba2858 (at 0@lo) reconnecting [ 2566.134961] Lustre: 99879:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2566.154533] LustreError: 99879:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2566.174919] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2566.267311] Lustre: lustre-MDT0000: new disk, initializing [ 2566.452227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2566.477277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2567.522797] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2567.538366] Lustre: 102305:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000008be7b7d9 x1849063074999424/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1763404173 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2569.206993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2569.221362] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2569.652053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2570.282535] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2594.276084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2594.281926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2594.292417] Lustre: Skipped 1 previous similar message [ 2599.169074] LustreError: 102825:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2599.171687] LustreError: 102825:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2599.283788] Lustre: server umount lustre-OST0000 complete [ 2604.515234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2604.516739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.534969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2604.545384] Lustre: Skipped 1 previous similar message [ 2604.549595] Lustre: Skipped 1 previous similar message [ 2608.252045] LustreError: 103027:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2608.256763] LustreError: 103027:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2608.282021] Lustre: server umount lustre-MDT0000 complete [ 2609.634696] LustreError: 102704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2609.649116] LustreError: 102704:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2617.494816] Lustre: server umount lustre-MDT0001 complete [ 2621.273431] Lustre: server umount MGS complete [ 2632.178211] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 13:30:21 (1763404221) [ 2645.605379] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 2647.747708] Key type lgssc unregistered [ 2648.017254] LNet: 104770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2648.025555] LNetError: 104770:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2648.040631] LNet: Removed LNI 192.168.203.130@tcp [ 2648.738390] Key type .llcrypt unregistered [ 2648.740903] Key type ._llcrypt unregistered [ 2662.867285] Key type ._llcrypt registered [ 2662.869231] Key type .llcrypt registered [ 2662.972310] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2663.882181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2664.082766] alg: No test for adler32 (adler32-zlib) [ 2665.127802] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 2665.339927] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 2667.047239] Key type lgssc registered [ 2668.230525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2675.890627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2675.908542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2677.460864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2681.801467] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2682.859297] LustreError: 105968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2682.871780] LustreError: 105968:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2687.971108] LustreError: 105968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2690.120187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2690.612447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2694.340230] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2699.645468] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2703.609690] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2710.085855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2710.517162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2716.005652] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2721.513560] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2722.803520] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2730.976629] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2730.979648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2730.991692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2733.027183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2733.031687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2733.035773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2735.553798] LustreError: 108214:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2735.766793] Lustre: server umount lustre-OST0000 complete [ 2741.729769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2741.733629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2741.743397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2741.746899] Lustre: Skipped 1 previous similar message [ 2745.843967] LustreError: 108416:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2745.847689] LustreError: 108416:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2745.975973] Lustre: server umount lustre-MDT0000 complete [ 2748.389330] LustreError: 105962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2748.413974] LustreError: 105962:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2749.243978] LustreError: 105947:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763404338 with bad export cookie 10042143165126052064 [ 2749.247525] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2749.249786] LustreError: 105947:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2749.360722] LustreError: 108618:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2749.364917] LustreError: 108618:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2749.495746] Lustre: server umount lustre-MDT0001 complete [ 2758.777382] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 2761.517166] Key type lgssc unregistered [ 2761.821536] LNet: 109149:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2761.834598] LNetError: 109149:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2761.862069] LNet: Removed LNI 192.168.203.130@tcp [ 2762.607639] Key type .llcrypt unregistered [ 2762.609472] Key type ._llcrypt unregistered [ 2777.201936] Key type ._llcrypt registered [ 2777.206604] Key type .llcrypt registered [ 2777.304490] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2778.132073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2778.222609] alg: No test for adler32 (adler32-zlib) [ 2779.278631] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 2779.477694] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 2781.135193] Key type lgssc registered [ 2782.090890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2787.846431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2787.859278] LustreError: 110211:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2787.874325] LustreError: 110213:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2787.877913] LustreError: 110211:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 2787.881315] LustreError: 110211:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2787.894947] LustreError: 110211:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2787.910500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2789.432687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2796.559706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2796.876229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2800.259085] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2804.778895] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing lsmod [ 2807.309975] LustreError: 111391:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2807.336536] LustreError: 111392:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2807.346539] LustreError: 111391:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4979 [ 2807.350149] LustreError: 111391:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2807.364865] LustreError: 111391:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2807.409217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2807.737496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2812.387228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2812.402222] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2812.414710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2816.349812] LustreError: 111579:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2816.558795] Lustre: server umount lustre-MDT0000 complete [ 2818.016091] LustreError: 110254:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2818.030811] LustreError: 110254:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2819.886963] LustreError: 110231:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763404409 with bad export cookie 14414863365717073766 [ 2819.893727] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2819.914947] LustreError: 110231:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2820.031949] LustreError: 111782:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2820.039295] LustreError: 111782:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2820.224972] Lustre: server umount lustre-MDT0001 complete [ 2833.733492] LustreError: 111982:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2833.741568] LustreError: 111982:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2833.935149] Lustre: server umount lustre-OST0000 complete [ 2841.748426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2842.310582] LustreError: 112459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2842.409734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2845.949432] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2847.714908] LustreError: 112459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2852.836948] LustreError: 112459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2854.193505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2854.777975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2858.122550] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2863.690140] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2866.987745] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2874.148336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2874.868276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2880.744122] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2885.779673] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2888.177963] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2895.336725] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2895.340409] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2895.359355] Lustre: Skipped 2 previous similar messages [ 2895.366235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2895.370944] Lustre: Skipped 2 previous similar messages [ 2898.405957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2898.406520] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2898.411475] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2903.529115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2903.532231] Lustre: Skipped 1 previous similar message [ 2908.642727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2908.646033] Lustre: Skipped 1 previous similar message [ 2909.151190] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2909.196797] LustreError: 114714:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2909.210570] LustreError: 114714:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2909.372110] Lustre: server umount lustre-OST0000 complete [ 2913.765450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2918.699863] LustreError: 114918:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2918.705022] LustreError: 114918:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2918.883996] LustreError: 113524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2918.916350] LustreError: 113524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2920.889503] Lustre: server umount lustre-MDT0000 complete [ 2924.743379] LustreError: 112439:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763404514 with bad export cookie 14414863365717074452 [ 2924.754659] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2924.768372] LustreError: 112439:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2925.111541] Lustre: server umount lustre-MDT0001 complete [ 2934.462788] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 2937.276577] Key type lgssc unregistered [ 2937.573626] LNet: 115653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2937.588367] LNetError: 115653:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2937.607061] LNet: Removed LNI 192.168.203.130@tcp [ 2938.480879] Key type .llcrypt unregistered [ 2938.484114] Key type ._llcrypt unregistered [ 2957.691428] Key type ._llcrypt registered [ 2957.692579] Key type .llcrypt registered [ 2957.781452] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 2958.518650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2958.534468] alg: No test for adler32 (adler32-zlib) [ 2959.555845] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 2959.765210] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 2961.431185] Key type lgssc registered [ 2962.336378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2968.684105] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2968.719776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2970.300759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2974.118856] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2979.817441] LustreError: 117032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2979.830478] LustreError: 117032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2981.858877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2982.245272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2985.784840] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 2990.054475] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2993.111761] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2999.947409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3000.354726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3005.825790] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3011.582248] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3014.710315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3020.275310] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3023.246175] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3026.913482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3026.921347] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3026.938658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3029.986996] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3029.988947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3030.066401] Lustre: Skipped 1 previous similar message [ 3032.243112] LustreError: 119379:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3032.390493] Lustre: server umount lustre-OST0000 complete [ 3038.692773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3038.695783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3038.710198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3041.778209] LustreError: 119584:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3041.782292] LustreError: 119584:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3041.862823] Lustre: server umount lustre-MDT0000 complete [ 3045.049326] LustreError: 117013:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763404634 with bad export cookie 18057828849706440486 [ 3045.059492] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3045.065410] LustreError: 117013:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3045.273441] LustreError: 119785:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3045.280176] LustreError: 119785:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3045.435293] Lustre: server umount lustre-MDT0001 complete [ 3054.448608] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 3057.264387] Key type lgssc unregistered [ 3057.592434] LNet: 120318:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3057.600439] LNetError: 120318:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3057.634185] LNet: Removed LNI 192.168.203.130@tcp [ 3058.433852] Key type .llcrypt unregistered [ 3058.435570] Key type ._llcrypt unregistered [ 3072.353771] Key type ._llcrypt registered [ 3072.355265] Key type .llcrypt registered [ 3072.486679] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 3073.507847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3073.534628] alg: No test for adler32 (adler32-zlib) [ 3074.571858] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 3074.817536] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 3076.503182] Key type lgssc registered [ 3077.624939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3084.471954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3084.501484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3086.046545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3089.311283] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3095.522448] LustreError: 121504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3095.565028] LustreError: 121504:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3096.799131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3097.121028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3100.431761] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3104.498980] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3107.483831] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3113.086884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3113.476603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3118.153621] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3122.354950] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3130.925551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3141.088230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3141.093438] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3141.108532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3141.112656] Lustre: Skipped 1 previous similar message [ 3144.708085] LustreError: 123769:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3144.925511] Lustre: server umount lustre-OST0000 complete [ 3148.256084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3148.264984] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3148.278104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3148.286017] Lustre: Skipped 1 previous similar message [ 3151.329992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3151.332160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3151.371691] Lustre: Skipped 1 previous similar message [ 3154.099318] LustreError: 123971:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3154.102333] LustreError: 123971:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3154.167251] Lustre: server umount lustre-MDT0000 complete [ 3156.450697] LustreError: 123120:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3156.460593] LustreError: 123120:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3157.257950] LustreError: 123972:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763404746 with bad export cookie 12244733221732479107 [ 3157.259873] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3157.297532] LustreError: 123972:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3157.506192] LustreError: 124174:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3157.509977] LustreError: 124174:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3157.697507] Lustre: server umount lustre-MDT0001 complete [ 3165.498815] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 3167.714806] Key type lgssc unregistered [ 3168.015778] LNet: 124707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3168.026854] LNetError: 124707:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3168.053424] LNet: Removed LNI 192.168.203.130@tcp [ 3168.678522] Key type .llcrypt unregistered [ 3168.681589] Key type ._llcrypt unregistered [ 3186.363622] Key type ._llcrypt registered [ 3186.365634] Key type .llcrypt registered [ 3186.450533] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 3187.304775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3187.311072] alg: No test for adler32 (adler32-zlib) [ 3188.403688] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 3188.619752] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 3190.319177] Key type lgssc registered [ 3191.472171] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3199.312877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3199.329827] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3200.813416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3204.228320] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3211.237895] LustreError: 126084:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3211.268306] LustreError: 126084:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3212.691443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3213.057012] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3216.575821] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3221.258984] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3224.498802] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3230.210614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3230.458564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3235.963887] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3241.866951] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3245.826840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3253.754809] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3257.477585] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3257.490139] Lustre: Skipped 2 previous similar messages [ 3261.249681] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3261.254942] Lustre: Skipped 2 previous similar messages [ 3263.056117] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3263.080957] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3270.308337] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.30@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3270.316619] Lustre: Skipped 2 previous similar messages [ 3271.960376] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.30@tcp 192.168.0.[2,10]@tcp [ 3277.427876] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.30/24@tcp in log lustre-MDT0000 [ 3277.433898] Lustre: Skipped 2 previous similar messages [ 3278.454228] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.30/24@tcp [ 3278.456941] Lustre: Skipped 1 previous similar message [ 3283.936405] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3283.948639] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3283.972982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3287.010577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3287.019422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3289.477982] LustreError: 129496:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3289.623382] Lustre: server umount lustre-OST0000 complete [ 3295.203326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3295.206945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3295.225448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3295.228399] Lustre: Skipped 1 previous similar message [ 3297.247770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3297.251626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3298.983625] LustreError: 129697:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3298.986714] LustreError: 129697:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3299.158664] Lustre: server umount lustre-MDT0000 complete [ 3302.059324] LustreError: 126064:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763404891 with bad export cookie 2040806123866844183 [ 3302.063873] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3302.071585] LustreError: 126064:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3302.224383] LustreError: 129899:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3302.240356] LustreError: 129899:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3302.435209] Lustre: server umount lustre-MDT0001 complete [ 3310.764481] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 3313.118714] Key type lgssc unregistered [ 3313.380524] LNet: 130429:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3313.392162] LNetError: 130429:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3313.416490] LNet: Removed LNI 192.168.203.130@tcp [ 3314.196891] Key type .llcrypt unregistered [ 3314.198915] Key type ._llcrypt unregistered [ 3337.282917] Key type ._llcrypt registered [ 3337.284642] Key type .llcrypt registered [ 3337.387763] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 3338.207262] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3338.217368] alg: No test for adler32 (adler32-zlib) [ 3339.400671] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 3339.609110] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 3341.255180] Key type lgssc registered [ 3342.290212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3349.624750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3349.667835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3351.326544] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3351.347645] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.30/24@tcp [ 3351.361258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3355.026178] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3360.745869] LustreError: 131951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3360.760455] LustreError: 131951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3363.132414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3363.457722] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3363.483392] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.30/24@tcp [ 3363.547318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3367.047384] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3371.962323] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3374.952420] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3380.331116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3380.731166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3385.118888] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3389.730806] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3396.147165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3407.851358] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3407.856625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3407.863041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3409.379500] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3409.384749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3409.401648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3411.429557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3411.448739] Lustre: Skipped 1 previous similar message [ 3413.376700] LustreError: 134274:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3413.469109] Lustre: server umount lustre-OST0000 complete [ 3416.546309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3416.552631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3421.665836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3421.668462] Lustre: Skipped 2 previous similar messages [ 3422.499596] LustreError: 134475:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3422.509449] LustreError: 134475:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3422.667198] Lustre: server umount lustre-MDT0000 complete [ 3425.399393] LustreError: 131931:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763405015 with bad export cookie 17127154325015738776 [ 3425.407563] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3425.501396] LustreError: 134677:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3425.504801] LustreError: 134677:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3425.619780] Lustre: server umount lustre-MDT0001 complete [ 3433.088579] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 3435.153893] Key type lgssc unregistered [ 3435.377224] LNet: 135209:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3435.381285] LNetError: 135209:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3435.406821] LNet: Removed LNI 192.168.203.130@tcp [ 3435.976265] Key type .llcrypt unregistered [ 3435.980983] Key type ._llcrypt unregistered [ 3456.644835] Key type ._llcrypt registered [ 3456.646517] Key type .llcrypt registered [ 3456.740823] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 3467.925946] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 3468.897494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3468.932705] alg: No test for adler32 (adler32-zlib) [ 3469.949335] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 3470.187238] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 3471.855223] Key type lgssc registered [ 3472.831731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3480.293893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3485.611993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3491.270791] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3496.448546] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3501.267556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3501.303931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3501.314624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3502.460060] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3502.492529] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3502.563687] Lustre: lustre-MDT0000: new disk, initializing [ 3502.648221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3502.680234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3505.257570] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3512.263164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3512.317894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3512.382561] Lustre: 138560:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3512.432085] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3512.435694] Lustre: Skipped 1 previous similar message [ 3512.485919] Lustre: lustre-MDT0001: new disk, initializing [ 3512.593649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3512.623609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3512.631810] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3515.270481] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3520.571659] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3522.875452] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3527.105653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3527.148746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3527.309389] Lustre: lustre-OST0000: new disk, initializing [ 3527.319316] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3527.365387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3531.615222] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3532.851547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3532.857256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3532.941685] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3536.932946] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3539.711486] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3539.904208] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3542.854145] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3543.029802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3548.643340] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3548.645160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3548.649034] LustreError: Skipped 1 previous similar message [ 3548.656775] Lustre: Skipped 1 previous similar message [ 3548.657560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3548.669807] Lustre: Skipped 1 previous similar message [ 3550.511877] LustreError: 141498:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3550.625944] Lustre: server umount lustre-OST0000 complete [ 3553.763144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3553.764422] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3553.770291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3553.775130] Lustre: Skipped 1 previous similar message [ 3558.883580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3558.888203] Lustre: Skipped 1 previous similar message [ 3559.074850] LustreError: 141700:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3559.078443] LustreError: 141700:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3559.161748] Lustre: server umount lustre-MDT0000 complete [ 3562.155587] LustreError: 138552:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763405151 with bad export cookie 1019762919678845190 [ 3562.159282] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3562.180752] LustreError: 138552:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3562.289716] LustreError: 141901:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3562.292978] LustreError: 141901:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3562.407335] Lustre: server umount lustre-MDT0001 complete [ 3568.268377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3568.614413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3571.278633] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3573.728774] LustreError: 142377:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3573.742466] LustreError: 142377:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3577.030390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3577.386606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3580.451605] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3584.216842] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3586.742123] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3591.179420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3591.399222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3595.668145] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3599.457163] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3602.279658] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3602.448358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3605.249991] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3605.396545] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3611.452093] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3611.494707] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3611.582420] Lustre: lustre-OST0001: new disk, initializing [ 3611.585611] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3611.654070] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3616.156928] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3618.385186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3618.405463] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3618.454465] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3623.093712] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3623.324435] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3626.648609] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3626.813687] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3629.617239] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa0904abc4000.ost_server_uuid 50 [ 3630.619744] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa0904abc4000.ost_server_uuid in FULL state after 0 sec [ 3639.267543] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3639.285468] Lustre: Skipped 1 previous similar message [ 3639.293164] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3639.296893] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3639.304684] Lustre: Skipped 1 previous similar message [ 3642.499090] LustreError: 146627:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3642.507254] LustreError: 146627:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3642.568300] Lustre: server umount lustre-OST0001 complete [ 3644.388912] LustreError: 143974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3644.414545] LustreError: 143974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3648.992167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3648.998427] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3649.022941] Lustre: Skipped 1 previous similar message [ 3649.031789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3649.509745] LustreError: 143974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3649.530142] LustreError: 143974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3651.701184] LustreError: 146829:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3651.703532] LustreError: 146829:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3651.827885] Lustre: server umount lustre-OST0000 complete [ 3654.625381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3654.649530] Lustre: Skipped 1 previous similar message [ 3659.232531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3659.243275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3659.245629] Lustre: Skipped 3 previous similar messages [ 3660.772262] LustreError: 147030:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3660.782046] LustreError: 147030:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3660.891850] Lustre: server umount lustre-MDT0000 complete [ 3664.543089] LustreError: 142358:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763405254 with bad export cookie 1019762919678846072 [ 3664.555051] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3664.559403] LustreError: 142358:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3664.862865] Lustre: server umount lustre-MDT0001 complete [ 3680.342211] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 3682.585837] Key type lgssc unregistered [ 3682.852918] LNet: 148221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3682.856432] LNetError: 148221:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3682.874251] LNet: Removed LNI 192.168.203.130@tcp [ 3683.566687] Key type .llcrypt unregistered [ 3683.574842] Key type ._llcrypt unregistered [ 3710.000141] Key type ._llcrypt registered [ 3710.002034] Key type .llcrypt registered [ 3710.101864] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 3723.934853] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 3724.586547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3724.633426] alg: No test for adler32 (adler32-zlib) [ 3725.676645] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 3725.924863] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 3727.655212] Key type lgssc registered [ 3728.794365] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3736.315208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3741.600695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3747.033146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3751.711028] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3756.686801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3756.735804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3756.743864] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3757.983395] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3758.021609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3758.103908] Lustre: lustre-MDT0000: new disk, initializing [ 3758.205989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3758.226814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3761.445457] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3769.281155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3769.324718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3769.390521] Lustre: 152168:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3769.433355] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3769.436510] Lustre: Skipped 1 previous similar message [ 3769.491778] Lustre: lustre-MDT0001: new disk, initializing [ 3769.579195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3769.614927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3769.628882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3772.358362] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3778.379105] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3781.469433] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3786.755653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3786.795813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3787.008769] Lustre: lustre-OST0000: new disk, initializing [ 3787.015610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3787.081094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3791.534670] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3793.020404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3793.033564] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3793.189197] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3798.053398] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3801.169521] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3801.310499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3804.514943] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3804.693108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3808.737163] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3808.740352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3808.751502] LustreError: Skipped 1 previous similar message [ 3808.755400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3808.767414] Lustre: Skipped 1 previous similar message [ 3808.781590] Lustre: Skipped 1 previous similar message [ 3812.348981] LustreError: 155108:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3812.524145] Lustre: server umount lustre-OST0000 complete [ 3815.908024] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3815.915981] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3815.924288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3818.977054] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3818.978608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3818.990604] Lustre: Skipped 1 previous similar message [ 3824.096965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3824.100238] Lustre: Skipped 1 previous similar message [ 3829.216993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3830.239159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3830.336914] LustreError: 155309:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3830.341362] LustreError: 155309:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3830.430925] Lustre: server umount lustre-MDT0000 complete [ 3833.470832] LustreError: 152950:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763405423 with bad export cookie 3178578036143747203 [ 3833.478100] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3833.480804] LustreError: 152950:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3833.624494] LustreError: 155511:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3833.640275] LustreError: 155511:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3833.828241] Lustre: server umount lustre-MDT0001 complete [ 3840.226616] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 13:50:29 (1763405429) [ 3853.069608] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 3860.602606] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 3868.793204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3874.175830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3878.831634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3884.491371] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3890.884731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3891.005802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3891.194248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3891.220496] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3891.275328] Lustre: lustre-MDT0000: new disk, initializing [ 3891.348585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3891.367052] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3895.172783] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3903.583474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3903.645712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3903.700443] Lustre: 158502:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3903.755704] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3903.759520] Lustre: Skipped 1 previous similar message [ 3903.812509] Lustre: lustre-MDT0001: new disk, initializing [ 3903.887778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3903.926733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3903.931307] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3907.397750] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3914.646381] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3918.257814] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3923.989237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3924.059558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3924.267175] Lustre: lustre-OST0000: new disk, initializing [ 3924.272522] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3924.350480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3928.888819] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3932.232664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3932.250849] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3932.362560] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3934.976806] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3938.758386] Lustre: Failing over lustre-OST0000 [ 3938.833418] LustreError: 160944:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3938.844118] LustreError: 160944:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3938.962761] Lustre: server umount lustre-OST0000 complete [ 3957.160738] Lustre: 150383:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763405530/real 1763405530] req@00000000a7c3376d x1849064491648896/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1763405546 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3957.191904] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3958.097942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3958.675731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3958.707723] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3959.647566] Lustre: 150385:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763405532/real 1763405532] req@000000006d534e2c x1849064491649280/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1763405548 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3959.687052] Lustre: 150385:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 3962.661686] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3963.722065] Lustre: 150383:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763405537/real 1763405537] req@0000000037804bf7 x1849064491650944/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763405553 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3964.728023] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3967.142191] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 3967.143895] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3968.257330] Lustre: Failing over lustre-MDT0000 [ 3968.421645] LustreError: 161926:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3968.436981] LustreError: 161926:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3968.528363] Lustre: server umount lustre-MDT0000 complete [ 3969.315805] Lustre: 150382:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763405542/real 1763405542] req@000000008a4b3768 x1849064491652096/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763405558 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3969.353757] Lustre: 150382:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3970.532684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3970.547965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3970.567831] Lustre: Skipped 1 previous similar message [ 3977.358551] LustreError: 158509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3977.376589] LustreError: 158509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3978.218076] LustreError: 158509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3978.248530] LustreError: 158509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3982.479881] LustreError: 158508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3987.207613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3987.297294] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3987.690175] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3991.528291] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 3992.716595] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3993.087416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3993.090422] Lustre: Skipped 1 previous similar message [ 3993.124739] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3993.182463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3998.691253] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3998.699050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3998.712665] Lustre: Skipped 2 previous similar messages [ 3998.726576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3998.730550] Lustre: Skipped 1 previous similar message [ 4000.590243] LustreError: 162810:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4000.604185] LustreError: 162810:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4000.718977] Lustre: server umount lustre-OST0000 complete [ 4008.417533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4010.613485] Lustre: server umount lustre-MDT0000 complete [ 4013.537817] LustreError: 158514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4013.571202] LustreError: 158514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4014.016902] LustreError: 158493:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763405603 with bad export cookie 3178578036143749317 [ 4014.026951] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4014.030351] LustreError: 158493:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4024.307585] LustreError: 163214:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4024.310878] LustreError: 163214:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4024.486783] Lustre: server umount lustre-MDT0001 complete [ 4032.923492] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 4035.121797] Key type lgssc unregistered [ 4035.377188] LNet: 163747:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4035.381926] LNetError: 163747:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4035.393925] LNet: Removed LNI 192.168.203.130@tcp [ 4035.950801] Key type .llcrypt unregistered [ 4035.953709] Key type ._llcrypt unregistered [ 4067.526364] Key type ._llcrypt registered [ 4067.528022] Key type .llcrypt registered [ 4067.588281] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 4068.215802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4068.231861] alg: No test for adler32 (adler32-zlib) [ 4069.238371] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 4069.466520] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 4071.118167] Key type lgssc registered [ 4072.086201] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4079.147581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4079.175409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4080.550950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4083.550494] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4089.192523] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4090.850495] LustreError: 165250:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4090.860130] LustreError: 165250:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4092.364989] LustreError: 165278:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4092.375057] LustreError: 165278:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4092.382138] LustreError: 165278:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 12, retries 0, failed: rc = -5 [ 4092.512362] LustreError: 165800:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4092.691740] Lustre: server umount lustre-MDT0000 complete [ 4104.244935] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 4111.610619] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 4119.596285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4124.783672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4130.002656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4135.078296] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4140.280475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4140.340685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4140.501758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4140.654425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4140.705963] Lustre: lustre-MDT0000: new disk, initializing [ 4140.781898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4140.803889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4143.738401] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4151.500735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4151.545901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4151.642364] Lustre: 168443:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4151.723079] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4151.725940] Lustre: Skipped 1 previous similar message [ 4151.835852] Lustre: lustre-MDT0001: new disk, initializing [ 4151.937843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4151.973216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4151.980943] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4155.431505] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4161.225339] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4163.904398] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4169.130360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4169.175957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4169.341875] Lustre: lustre-OST0000: new disk, initializing [ 4169.345248] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4169.403033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4173.486253] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4176.410263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4176.424072] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4176.502219] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4180.341400] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4184.585083] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4185.972655] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 13:56:15 (1763405775) [ 4381.155513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4381.158797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4381.177641] Lustre: Skipped 2 previous similar messages [ 4381.181236] Lustre: Skipped 1 previous similar message [ 4384.448834] LustreError: 171964:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4384.459205] LustreError: 171964:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4384.737097] Lustre: server umount lustre-MDT0000 complete [ 4386.275867] LustreError: 169780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4386.323892] LustreError: 169780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4388.601349] LustreError: 168434:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763405978 with bad export cookie 6531557483828693560 [ 4388.605261] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4388.619443] LustreError: 168434:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4391.392663] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4391.393980] LustreError: 171854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4391.418967] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4391.421756] Lustre: Skipped 1 previous similar message [ 4393.442754] LustreError: 171854:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4393.445986] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4394.784509] LustreError: 172170:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4394.787350] LustreError: 172170:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4394.851155] Lustre: server umount lustre-MDT0001 complete [ 4404.526585] LustreError: 172374:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4404.531819] LustreError: 172374:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4404.661793] Lustre: server umount lustre-OST0000 complete [ 4412.910736] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 4421.831201] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 4429.811029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4435.332773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4440.723417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4446.544132] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4453.323596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4453.394960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4453.734937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4453.803230] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4453.954436] Lustre: lustre-MDT0000: new disk, initializing [ 4454.022406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4454.036804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4457.761346] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4467.316302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4467.355689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4467.423773] Lustre: 174714:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4467.464730] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4467.468051] Lustre: Skipped 1 previous similar message [ 4467.516987] Lustre: lustre-MDT0001: new disk, initializing [ 4467.584499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4467.618055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4467.629883] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4470.467530] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4476.396474] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4479.251695] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4484.900762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4484.968719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4485.151876] Lustre: lustre-OST0000: new disk, initializing [ 4485.159957] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4485.221067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4490.239978] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4491.319175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4491.339071] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4491.495309] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4497.435062] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4501.103734] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4501.345954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4505.184930] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4505.391357] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4507.104522] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4507.109493] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4507.124358] LustreError: Skipped 1 previous similar message [ 4507.145759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4507.150452] Lustre: Skipped 1 previous similar message [ 4512.230464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4513.085157] LustreError: 177662:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4513.102842] LustreError: 177662:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4513.190802] Lustre: server umount lustre-OST0000 complete [ 4517.344435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4517.382069] Lustre: Skipped 1 previous similar message [ 4522.465251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4522.483797] Lustre: Skipped 4 previous similar messages [ 4531.167287] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4531.263748] LustreError: 177864:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4531.269448] LustreError: 177864:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4531.431898] Lustre: server umount lustre-MDT0000 complete [ 4532.707148] LustreError: 175675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4532.722743] LustreError: 175675:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4535.031861] LustreError: 174706:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763406124 with bad export cookie 6531557483828852390 [ 4535.037897] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4535.363147] Lustre: server umount lustre-MDT0001 complete [ 4543.018467] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:02:12 (1763406132) [ 4555.872238] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 4563.206942] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 4570.202275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4576.035136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4581.194588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4586.334513] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4593.197483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4593.274377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4593.507874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4593.547687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4593.690103] Lustre: lustre-MDT0000: new disk, initializing [ 4593.802113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4593.832792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4597.724242] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4607.601421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4607.688167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4607.834085] Lustre: 181061:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4608.227923] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4612.323332] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4619.161637] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4622.443304] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4628.520459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4628.636886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4629.034440] Lustre: lustre-OST0000: new disk, initializing [ 4629.042163] Lustre: Skipped 1 previous similar message [ 4629.050213] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4629.053230] Lustre: Skipped 2 previous similar messages [ 4634.666793] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4637.779114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4637.787170] Lustre: Skipped 1 previous similar message [ 4637.793797] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4637.905327] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4642.011742] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4648.416043] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4648.422303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4648.432887] Lustre: Skipped 1 previous similar message [ 4648.437261] LustreError: Skipped 1 previous similar message [ 4648.445933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4648.453557] Lustre: Skipped 2 previous similar messages [ 4654.409205] LustreError: 183607:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4654.412293] LustreError: 183607:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4654.545390] Lustre: server umount lustre-OST0000 complete [ 4658.661486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4658.677454] Lustre: Skipped 2 previous similar messages [ 4664.011072] Lustre: server umount lustre-MDT0000 complete [ 4667.460822] LustreError: 181846:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763406257 with bad export cookie 6531557483828853272 [ 4667.471909] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4667.481332] LustreError: 181846:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4667.770341] Lustre: server umount lustre-MDT0001 complete [ 4674.799839] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 14:04:23 (1763406263) [ 4688.100523] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 4695.382786] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 4703.132180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4708.485274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4713.442876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4718.904193] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4725.189235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4725.274508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4725.538932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4725.576109] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4725.712483] Lustre: lustre-MDT0000: new disk, initializing [ 4725.833738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4725.838706] Lustre: Skipped 2 previous similar messages [ 4725.858138] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4729.387048] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4740.151765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4740.199908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4740.317382] Lustre: 187001:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4740.675117] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4745.172499] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4753.944795] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4757.553671] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4763.757561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4763.850217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4769.455387] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4770.554053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4776.319126] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4783.670182] LustreError: 189548:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4783.680392] LustreError: 189548:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4783.804485] Lustre: server umount lustre-OST0000 complete [ 4791.269788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4791.279762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4791.281335] Lustre: Skipped 1 previous similar message [ 4791.300772] Lustre: Skipped 6 previous similar messages [ 4797.526756] Lustre: server umount lustre-MDT0000 complete [ 4801.010305] LustreError: 186994:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763406390 with bad export cookie 6531557483828854357 [ 4801.020380] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4811.355461] Lustre: server umount lustre-MDT0001 complete [ 4822.039842] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 4826.002484] Key type lgssc unregistered [ 4826.456770] LNet: 190484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4826.478635] LNetError: 190484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4826.493321] LNet: Removed LNI 192.168.203.130@tcp [ 4827.187632] Key type .llcrypt unregistered [ 4827.193844] Key type ._llcrypt unregistered [ 4847.451874] Key type ._llcrypt registered [ 4847.455015] Key type .llcrypt registered [ 4847.543024] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 4848.393969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4848.468411] alg: No test for adler32 (adler32-zlib) [ 4849.489985] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 4849.706839] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 4851.391175] Key type lgssc registered [ 4852.349678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4858.139983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4858.150462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4859.506494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4862.523690] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4869.518694] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4869.870934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4873.509804] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4878.156731] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4881.787023] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4888.509694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4888.825416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4893.692343] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4898.215364] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4906.234829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4925.922860] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4925.932298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4925.959097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4926.950028] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4926.957191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4926.982675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4926.995362] Lustre: Skipped 1 previous similar message [ 4930.291505] LustreError: 194129:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4930.364556] Lustre: server umount lustre-OST0000 complete [ 4936.675397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4936.678573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4936.695663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4939.493924] LustreError: 194333:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4939.511341] LustreError: 194333:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4939.584601] Lustre: server umount lustre-MDT0000 complete [ 4942.304658] LustreError: 191859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4942.328740] LustreError: 191859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4942.871145] LustreError: 191843:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763406532 with bad export cookie 4538375665113029190 [ 4942.876480] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4942.893258] LustreError: 191843:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4943.038071] LustreError: 194534:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4943.067799] LustreError: 194534:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4943.228544] Lustre: server umount lustre-MDT0001 complete [ 4951.504337] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 4953.882742] Key type lgssc unregistered [ 4954.116432] LNet: 195065:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4954.124914] LNetError: 195065:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4955.178614] LNet: Removed LNI 192.168.203.130@tcp [ 4955.787358] Key type .llcrypt unregistered [ 4955.789478] Key type ._llcrypt unregistered [ 4974.283236] Key type ._llcrypt registered [ 4974.288221] Key type .llcrypt registered [ 4974.403755] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 4975.270783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4975.312122] alg: No test for adler32 (adler32-zlib) [ 4976.435581] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 4976.640987] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 4978.287157] Key type lgssc registered [ 4979.321762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4986.000980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4986.043703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4987.683159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4991.709316] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 4998.116061] LustreError: 196441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4998.146136] LustreError: 196441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5000.836704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5001.402074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5005.993843] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5011.437408] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5014.855789] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5020.708749] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5020.989424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5025.694677] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5030.092814] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5033.261615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5041.826957] Lustre: lustre-OST0000: Not available for connect from 192.168.203.30@tcp (stopping) [ 5042.656063] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5042.665540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5042.680388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5043.681888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5043.684477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5043.689242] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5043.704326] Lustre: Skipped 1 previous similar message [ 5046.289578] LustreError: 198689:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5046.485173] Lustre: server umount lustre-OST0000 complete [ 5051.747349] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffffa090475c3800.ost_server_uuid 50 [ 5053.543346] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5067.745463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5067.756481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5067.766385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5072.483565] LustreError: 199061:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5072.488668] LustreError: 199061:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5072.572445] Lustre: server umount lustre-MDT0000 complete [ 5073.895200] LustreError: 196436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5073.928761] LustreError: 196436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5075.952638] LustreError: 196421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763406665 with bad export cookie 383490763241207020 [ 5075.958720] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5075.963965] LustreError: 196421:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5076.103685] LustreError: 199263:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5076.109184] LustreError: 199263:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5076.328427] Lustre: server umount lustre-MDT0001 complete [ 5084.173875] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 14:11:13 (1763406673) [ 5090.530867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5090.807855] LustreError: 199933:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5090.869169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5095.388962] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5095.907478] LustreError: 199933:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5101.029302] LustreError: 199933:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5103.359331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5103.773695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5107.213802] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5111.747075] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5114.894822] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5120.695623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5121.088693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5125.720159] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5130.837854] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5135.412726] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5136.486424] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5136.569327] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5136.726932] Lustre: lustre-OST0001: new disk, initializing [ 5136.739772] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5141.731823] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5144.677712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5144.691340] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5144.762247] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5148.340755] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5150.689228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5150.693205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5150.702123] Lustre: Skipped 1 previous similar message [ 5150.706728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5150.710783] Lustre: Skipped 2 previous similar messages [ 5154.783988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5154.787868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5156.549130] LustreError: 203136:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5156.553938] LustreError: 203136:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5156.598316] Lustre: server umount lustre-OST0000 complete [ 5159.053204] LustreError: 201531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5160.810109] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5160.971452] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5163.956505] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5164.124441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5175.778775] LustreError: 201531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5175.801171] LustreError: 201531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5180.899265] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5180.904104] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5180.908048] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5180.908057] Lustre: Skipped 4 previous similar messages [ 5180.916965] Lustre: Skipped 1 previous similar message [ 5180.952039] LustreError: Skipped 1 previous similar message [ 5183.133813] LustreError: 203861:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5183.142815] LustreError: 203861:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5183.346092] Lustre: server umount lustre-OST0001 complete [ 5193.418272] LustreError: 204062:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5193.436871] LustreError: 204062:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5193.565932] Lustre: server umount lustre-MDT0000 complete [ 5196.978740] LustreError: 199915:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763406786 with bad export cookie 383490763241208091 [ 5196.988482] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5196.990036] LustreError: 199915:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5197.343239] Lustre: server umount lustre-MDT0001 complete [ 5202.368449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5205.349026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5208.160543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5211.429411] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5218.118229] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 14:13:26 (1763406806) [ 5225.472736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5225.970368] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5226.047660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5226.356503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5226.358807] Lustre: Skipped 1 previous similar message [ 5230.437756] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5240.878396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5241.016454] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5241.069127] Lustre: 205756:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5245.691977] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5251.049860] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5254.329342] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5260.299962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5260.576900] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5260.768795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5260.775321] Lustre: Skipped 1 previous similar message [ 5266.121718] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5267.615923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5271.702217] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5279.307692] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5279.428814] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5285.127286] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5290.507133] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5293.545154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5293.548777] LustreError: Skipped 1 previous similar message [ 5293.550886] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5293.586227] Lustre: Skipped 2 previous similar messages [ 5293.598800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5293.601702] Lustre: Skipped 4 previous similar messages [ 5299.396196] LustreError: 208957:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5299.403775] LustreError: 208957:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5299.497914] Lustre: server umount lustre-OST0000 complete [ 5301.390944] LustreError: 207387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5301.424557] LustreError: 207387:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5320.467956] Lustre: server umount lustre-OST0001 complete [ 5327.844659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5327.846653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5327.853114] Lustre: Skipped 4 previous similar messages [ 5327.861085] Lustre: Skipped 6 previous similar messages [ 5330.222246] Lustre: server umount lustre-MDT0000 complete [ 5333.705983] LustreError: 205748:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763406923 with bad export cookie 383490763241209491 [ 5333.710602] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5333.940298] LustreError: 209584:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5333.953393] LustreError: 209584:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 5334.303665] Lustre: server umount lustre-MDT0001 complete [ 5339.572466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5343.755537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5347.525465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5351.192461] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5358.059097] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 14:15:47 (1763406947) [ 5372.163477] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 5379.982135] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 5387.571474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5393.173452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5399.640296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5405.333819] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5410.822727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5410.891127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5411.101913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5411.123930] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5411.193485] Lustre: lustre-MDT0000: new disk, initializing [ 5411.273201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5411.278993] Lustre: Skipped 1 previous similar message [ 5411.297555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5414.345846] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5423.112298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5423.175638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5423.282663] Lustre: 213409:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5423.343305] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5423.345830] Lustre: Skipped 1 previous similar message [ 5423.401846] Lustre: lustre-MDT0001: new disk, initializing [ 5423.543747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5423.556503] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5426.637290] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5432.994928] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5436.161326] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5442.216916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5442.306301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5442.538904] Lustre: lustre-OST0000: new disk, initializing [ 5442.550177] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5447.756842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5447.778683] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5448.030302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5449.174737] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5457.722660] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5462.948728] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5463.127433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5467.355667] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5467.664151] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5473.761891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5473.766561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5473.795324] LustreError: Skipped 2 previous similar messages [ 5473.799095] Lustre: Skipped 1 previous similar message [ 5473.807348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5475.991678] LustreError: 216348:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5475.996409] LustreError: 216348:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 5476.216922] Lustre: server umount lustre-OST0000 complete [ 5486.597113] Lustre: server umount lustre-MDT0000 complete [ 5489.119870] LustreError: 214679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5489.150891] LustreError: 214679:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 5489.954371] LustreError: 214190:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763407079 with bad export cookie 383490763241211045 [ 5489.959074] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5489.978656] LustreError: 214190:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5490.213196] Lustre: server umount lustre-MDT0001 complete [ 5497.478414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5500.833536] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5507.405761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5510.522125] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5514.823916] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5517.829384] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5523.329399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5528.053598] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5532.086826] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5535.013938] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5535.139876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5538.175425] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5538.340700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5540.320259] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5545.878352] Lustre: server umount lustre-OST0000 complete [ 5550.740241] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5554.461406] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5582.632286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5582.882469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5582.894330] Lustre: Skipped 5 previous similar messages [ 5583.915548] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5583.932332] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5587.119684] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5591.751896] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5594.146688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5594.164033] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5601.945028] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 8 sec [ 5609.442406] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5609.452070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5609.474741] Lustre: Skipped 5 previous similar messages [ 5609.477826] Lustre: Skipped 6 previous similar messages [ 5612.225289] LustreError: 221613:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5612.237820] LustreError: 221613:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 5612.396094] Lustre: server umount lustre-OST0000 complete [ 5623.940117] Lustre: server umount lustre-MDT0000 complete [ 5624.802074] LustreError: 217229:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5624.823855] LustreError: 217229:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5627.142130] LustreError: 218848:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763407216 with bad export cookie 383490763241211920 [ 5627.148370] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5627.159556] LustreError: 218848:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5627.457225] Lustre: server umount lustre-MDT0001 complete [ 5634.563959] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 14:20:23 (1763407223) [ 5640.058216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5643.541685] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5651.823539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5655.813207] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5660.882906] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5664.574623] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5671.153990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5677.651764] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5682.353837] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5686.207947] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5686.394681] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5690.557032] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5690.768985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5697.249361] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5697.354312] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5697.492473] Lustre: lustre-OST0001: new disk, initializing [ 5697.495665] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5703.296838] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5703.330047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5703.334576] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5703.467368] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5711.082606] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5715.395675] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5715.591149] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5719.463812] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5719.721233] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5724.128340] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5724.147854] LustreError: Skipped 4 previous similar messages [ 5727.542787] Lustre: server umount lustre-OST0001 complete [ 5734.427890] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5734.757853] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5740.466025] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5740.759873] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5769.860104] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5775.333532] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5775.364741] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5775.379814] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5776.340556] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5783.403673] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5785.188713] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5793.238646] Lustre: server umount lustre-OST0001 complete [ 5815.075487] Lustre: server umount lustre-OST0000 complete [ 5824.772632] Lustre: server umount lustre-MDT0000 complete [ 5828.107393] LustreError: 222668:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763407417 with bad export cookie 383490763241212886 [ 5828.113786] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5828.121792] LustreError: 222668:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5834.750144] Lustre: server umount lustre-MDT0001 complete [ 5840.861033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5845.178774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5849.075352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5853.049896] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5860.618696] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 14:24:09 (1763407449) [ 5866.668313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5867.045147] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5867.116669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5867.291245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5867.298706] Lustre: Skipped 5 previous similar messages [ 5871.609863] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5880.834817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5880.971254] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5880.987786] Lustre: 230676:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5885.482997] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5891.492993] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5895.242804] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5902.188388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5902.474484] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5908.796723] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5914.187607] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5922.490145] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5922.578687] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5927.918610] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 5932.658234] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5933.909236] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5937.391539] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5937.570636] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5940.821317] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5941.041980] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5943.772402] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa09049bc2800.ost_server_uuid 50 [ 5945.154103] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa09049bc2800.ost_server_uuid in FULL state after 0 sec [ 5945.954662] Lustre: Permanently deactivating lustre-OST0001 [ 5945.972859] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5948.845308] Lustre: Permanently reactivating lustre-OST0001 [ 5953.400499] LustreError: 234666:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5953.406976] LustreError: 234666:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 5953.485200] Lustre: server umount lustre-OST0001 complete [ 5955.565643] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5955.576176] LustreError: 232300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5955.618933] Lustre: Skipped 11 previous similar messages [ 5955.621720] LustreError: 232300:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 35 previous similar messages [ 5959.142334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5959.152735] Lustre: Skipped 14 previous similar messages [ 5963.164948] Lustre: server umount lustre-OST0000 complete [ 5973.133049] Lustre: server umount lustre-MDT0000 complete [ 5976.978103] LustreError: 230667:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763407566 with bad export cookie 383490763241214167 [ 5976.983430] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5977.003235] LustreError: 230667:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5977.546767] Lustre: server umount lustre-MDT0001 complete [ 5982.487390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5985.773647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5989.508765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5993.079207] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6001.299095] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 14:26:29 (1763407589) [ 6002.264384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6008.572473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6008.798832] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6008.833692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6008.841195] Lustre: Skipped 5 previous similar messages [ 6013.337974] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6022.299181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6022.458434] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6022.488700] Lustre: 236826:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6026.759795] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6033.430394] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6037.258947] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6043.792832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6043.871623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6044.130446] Lustre: Permanently deactivating lustre-OST0000 [ 6044.155856] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6050.051408] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6051.859145] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6056.335634] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6064.330885] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6064.402295] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6064.404758] Lustre: Skipped 1 previous similar message [ 6071.795701] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6072.519240] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6077.213566] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6079.178739] Lustre: Permanently reactivating lustre-OST0000 [ 6079.205495] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6079.213985] Lustre: Skipped 2 previous similar messages [ 6086.816314] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6086.838127] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 35 seconds. I think it's dead, and I am evicting it. exp 000000009f5f3bf7, cur 1763407676 deadline 1763407671 last 1763407641 [ 6086.901256] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6097.282555] Lustre: server umount lustre-OST0001 complete [ 6098.413912] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6098.433133] LustreError: Skipped 9 previous similar messages [ 6107.663846] Lustre: server umount lustre-OST0000 complete [ 6117.457514] Lustre: server umount lustre-MDT0000 complete [ 6120.960941] LustreError: 236816:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763407710 with bad export cookie 383490763241215819 [ 6120.970573] LustreError: 236816:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6120.971507] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6121.388825] Lustre: server umount lustre-MDT0001 complete [ 6131.365103] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 6133.758119] Key type lgssc unregistered [ 6134.126559] LNet: 241244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6134.141415] LNetError: 241244:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6134.166676] LNet: Removed LNI 192.168.203.130@tcp [ 6134.966207] Key type .llcrypt unregistered [ 6134.967813] Key type ._llcrypt unregistered [ 6158.752763] Key type ._llcrypt registered [ 6158.754534] Key type .llcrypt registered [ 6158.866603] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 6159.867163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6159.884983] alg: No test for adler32 (adler32-zlib) [ 6161.167107] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 6161.397887] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 6163.063906] Key type lgssc registered [ 6164.255772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6167.532699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6174.128449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6174.149225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6175.848158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6180.071309] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6185.445666] LustreError: 242694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6185.453216] LustreError: 242694:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6188.972474] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6189.051879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6189.261928] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6189.305734] Lustre: 242684:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6189.317736] Lustre: Permanently deactivating lustre-MDT0001 [ 6189.335341] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6189.656190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6190.323032] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6190.331682] LustreError: 242725:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6190.353065] LustreError: 242725:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6190.373807] LustreError: 242725:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 6194.946594] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6202.365499] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6206.974784] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6216.209918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6216.685754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6220.777552] LustreError: 244315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6220.789987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6223.355505] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6225.901584] LustreError: 244314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6228.944904] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6231.024393] LustreError: 244314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6235.109972] LustreError: 244314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6235.139797] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6235.145877] LustreError: 244314:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6235.793447] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6236.150207] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6241.313122] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6241.316893] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6242.101898] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6248.253244] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6252.345718] Lustre: Permanently reactivating lustre-MDT0001 [ 6252.358732] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6252.362771] Lustre: Skipped 2 previous similar messages [ 6256.834637] Lustre: lustre-MDT0001: Client 30450f00-0dbe-4cc2-ae65-9cce84a0341b (at 192.168.203.30@tcp) reconnecting [ 6262.046681] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6264.169150] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6269.810782] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6270.098419] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6274.517694] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6274.689688] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6277.282924] Lustre: Permanently deactivating lustre-MDT0001 [ 6277.299633] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6277.302201] Lustre: Skipped 2 previous similar messages [ 6285.366558] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6297.056240] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6297.058573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6297.070224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6297.571419] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6297.574126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6297.600898] Lustre: Skipped 2 previous similar messages [ 6297.622634] Lustre: Skipped 2 previous similar messages [ 6300.425712] LustreError: 246818:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6300.626495] Lustre: server umount lustre-MDT0000 complete [ 6302.687941] LustreError: 242695:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6302.720557] LustreError: 242695:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6304.467310] LustreError: 246613:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763407894 with bad export cookie 11295548068446948387 [ 6304.471197] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6304.477189] LustreError: 246613:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6304.635466] LustreError: 247019:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 6304.641653] LustreError: 247019:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6304.835540] Lustre: server umount lustre-MDT0001 complete [ 6310.587502] LustreError: 247220:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6310.597260] LustreError: 247220:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6310.712322] Lustre: server umount lustre-OST0000 complete [ 6322.745158] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 14:31:51 (1763407911) [ 6334.985577] LustreError: 248226:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6334.988712] LustreError: 248226:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6335.208415] Lustre: server umount lustre-OST0001 complete [ 6342.217897] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 6352.585307] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 6362.206576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6368.383740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6374.096227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6380.552786] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6387.540051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6387.590592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6387.784724] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6387.793647] Lustre: Skipped 2 previous similar messages [ 6387.828929] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6387.993781] Lustre: lustre-MDT0000: new disk, initializing [ 6388.084314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6388.106746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6392.446498] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6402.275060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6402.365407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6402.509886] Lustre: 250412:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6402.578941] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6402.583022] Lustre: Skipped 1 previous similar message [ 6402.767667] Lustre: lustre-MDT0001: new disk, initializing [ 6402.886711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6402.959626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6402.974792] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6408.266969] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6416.204242] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6419.643970] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6427.038474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6427.153686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6427.558282] Lustre: lustre-OST0000: new disk, initializing [ 6427.564792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6427.689981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6433.091588] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6434.340693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6434.350805] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6434.489211] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6440.225634] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6442.280260] LustreError: 250421:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6451.337653] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6451.478957] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6451.663835] Lustre: lustre-OST0001: new disk, initializing [ 6451.667647] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6452.314502] LustreError: 250421:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6452.339331] LustreError: 250421:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6457.570200] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6459.498259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6459.504141] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6459.621970] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6462.363055] LustreError: 250421:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6465.805502] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6467.643204] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6470.128398] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6470.131421] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6470.133119] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6470.163660] Lustre: Skipped 1 previous similar message [ 6470.198126] LustreError: Skipped 1 previous similar message [ 6475.232429] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6475.244152] Lustre: Skipped 2 previous similar messages [ 6475.971471] LustreError: 253909:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6475.990838] LustreError: 253909:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6476.113095] Lustre: server umount lustre-OST0001 complete [ 6480.360582] LustreError: 253253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6480.394915] LustreError: 253253:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6495.715599] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6495.718649] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6495.726585] Lustre: Skipped 1 previous similar message [ 6495.739886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6495.743959] Lustre: Skipped 1 previous similar message [ 6499.034504] LustreError: 254134:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6499.052692] LustreError: 254134:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6499.308415] Lustre: server umount lustre-OST0000 complete [ 6505.440241] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6505.455899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6505.489159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6510.000777] Lustre: server umount lustre-MDT0000 complete [ 6513.836111] LustreError: 250406:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763408103 with bad export cookie 11295548068446951201 [ 6513.842867] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6513.861879] LustreError: 250406:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6520.152086] LustreError: 254538:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 6520.155751] LustreError: 254538:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 6520.408468] Lustre: server umount lustre-MDT0001 complete [ 6531.936370] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 6534.978438] Key type lgssc unregistered [ 6535.390051] LNet: 255070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6535.411121] LNetError: 255070:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6535.483226] LNet: Removed LNI 192.168.203.130@tcp [ 6536.313754] Key type .llcrypt unregistered [ 6536.315917] Key type ._llcrypt unregistered [ 6573.627990] Key type ._llcrypt registered [ 6573.638717] Key type .llcrypt registered [ 6573.766414] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 6588.845874] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 6589.904255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6590.080344] alg: No test for adler32 (adler32-zlib) [ 6591.326196] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 6591.700856] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 6593.407490] Key type lgssc registered [ 6594.774532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6604.580041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6611.084680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6617.985308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6625.258757] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6633.042550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6633.106881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6633.131495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6634.410534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6634.456265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6634.557704] Lustre: lustre-MDT0000: new disk, initializing [ 6634.713691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6634.753763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6639.262460] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6649.039185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6649.119755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6649.200453] Lustre: 259024:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6649.285253] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6649.296934] Lustre: Skipped 1 previous similar message [ 6649.452774] Lustre: lustre-MDT0001: new disk, initializing [ 6649.575532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6649.606370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6649.621193] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6653.184754] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6660.893829] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6664.592937] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6670.893402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6670.949283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6671.158770] Lustre: lustre-OST0000: new disk, initializing [ 6671.163921] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6671.257289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6676.715588] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6680.143692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6680.148198] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6680.259220] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6683.458448] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6687.341416] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6687.574861] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6691.395056] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6691.557942] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6695.906080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6695.911831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6695.942951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6696.009767] Lustre: Skipped 1 previous similar message [ 6699.278873] LustreError: 261965:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6699.377378] Lustre: server umount lustre-OST0000 complete [ 6706.144967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6706.150698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6706.165359] Lustre: Skipped 1 previous similar message [ 6706.175409] Lustre: Skipped 2 previous similar messages [ 6708.972551] LustreError: 262167:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6708.986603] LustreError: 262167:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6709.179087] Lustre: server umount lustre-MDT0000 complete [ 6712.653249] LustreError: 260772:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763408302 with bad export cookie 16207236561117419116 [ 6712.655232] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6712.660951] LustreError: 260772:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6712.812528] LustreError: 262368:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6712.815335] LustreError: 262368:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6713.057444] Lustre: server umount lustre-MDT0001 complete [ 6722.182764] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 14:38:30 (1763408310) [ 6728.632993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6729.252645] LustreError: 263040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6729.297601] LustreError: 263040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6729.384133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6732.863148] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6734.817228] LustreError: 263040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6738.917579] LustreError: 263040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6741.157120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6741.569600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6745.290741] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6750.325359] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6753.810859] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6760.655943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6760.989495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6765.641733] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6770.173284] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6825.443425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6825.446611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6825.455261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6825.476960] Lustre: Skipped 1 previous similar message [ 6830.568608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6830.572588] Lustre: Skipped 2 previous similar messages [ 6831.132571] LustreError: 265366:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6831.143408] LustreError: 265366:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6831.286369] Lustre: server umount lustre-OST0000 complete [ 6835.166334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6847.897021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6848.323407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6853.210711] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6855.537884] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6855.576526] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6855.582514] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6855.609964] Lustre: Skipped 1 previous similar message [ 6859.180286] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6901.745327] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6901.745980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6901.756221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6901.768535] Lustre: Skipped 1 previous similar message [ 6904.752170] LustreError: 267031:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6904.764463] LustreError: 267031:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6905.018634] Lustre: server umount lustre-OST0000 complete [ 6910.432842] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6910.452572] LustreError: Skipped 1 previous similar message [ 6910.477135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6910.520340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6910.530367] Lustre: Skipped 1 previous similar message [ 6916.259610] LustreError: 267235:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6916.263973] LustreError: 267235:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6916.370164] Lustre: server umount lustre-MDT0000 complete [ 6917.090262] LustreError: 263036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6917.102083] LustreError: 263036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6919.622698] LustreError: 264661:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763408509 with bad export cookie 16207236561117419998 [ 6919.627569] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6919.643713] LustreError: 264661:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6919.893900] Lustre: server umount lustre-MDT0001 complete [ 6929.112338] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 6931.803826] Key type lgssc unregistered [ 6932.135064] LNet: 267968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6932.149092] LNetError: 267968:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6932.165194] LNet: Removed LNI 192.168.203.130@tcp [ 6933.041560] Key type .llcrypt unregistered [ 6933.051684] Key type ._llcrypt unregistered [ 6956.834727] Key type ._llcrypt registered [ 6956.839791] Key type .llcrypt registered [ 6956.958231] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 6957.936799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6958.053153] alg: No test for adler32 (adler32-zlib) [ 6959.080920] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 6959.400385] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 6961.072150] Key type lgssc registered [ 6962.209869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6969.995779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6970.031281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6971.740433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6975.921619] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6982.123170] LustreError: 269363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6982.137715] LustreError: 269363:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6984.305197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6984.570987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6988.125292] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 6993.364616] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6996.432124] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7004.820544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7005.511659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7012.150274] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7016.831832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7017.570968] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7042.534141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7042.536386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7042.543676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7042.545416] Lustre: Skipped 1 previous similar message [ 7047.618112] LustreError: 272381:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7047.659410] LustreError: 270962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7047.799729] Lustre: server umount lustre-OST0000 complete [ 7051.240914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7051.255278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7051.288164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7051.290966] Lustre: Skipped 1 previous similar message [ 7052.769693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7052.773531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7057.346825] LustreError: 272582:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7057.379168] LustreError: 272582:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7057.586104] Lustre: server umount lustre-MDT0000 complete [ 7057.890202] LustreError: 269360:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7057.982325] LustreError: 269360:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7061.945091] LustreError: 269344:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763408651 with bad export cookie 13381466553942928103 [ 7061.959268] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7062.201840] LustreError: 272784:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7062.225436] LustreError: 272784:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7062.518670] Lustre: server umount lustre-MDT0001 complete [ 7074.390743] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 7077.053147] Key type lgssc unregistered [ 7077.364578] LNet: 273317:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7077.372474] LNetError: 273317:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7078.437718] LNet: Removed LNI 192.168.203.130@tcp [ 7079.170981] Key type .llcrypt unregistered [ 7079.173127] Key type ._llcrypt unregistered [ 7093.614211] Key type ._llcrypt registered [ 7093.615760] Key type .llcrypt registered [ 7093.728206] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 7094.810639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7094.819892] alg: No test for adler32 (adler32-zlib) [ 7095.843731] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 7096.033619] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 7097.695138] Key type lgssc registered [ 7098.883810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7105.821208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7105.867876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7107.353478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7112.255908] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7117.792808] LustreError: 274498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7117.802076] LustreError: 274498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7119.384395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7119.630157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7122.758963] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7128.606798] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7131.701547] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7138.572928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7139.007694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7144.436136] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7149.924176] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7153.318921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7165.920281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7165.932540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7165.950623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7169.002404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7169.002880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7169.008734] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7169.023037] Lustre: Skipped 1 previous similar message [ 7171.455219] LustreError: 276891:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7171.600509] Lustre: server umount lustre-OST0000 complete [ 7176.163113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7176.166616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7176.197571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7179.233484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7179.235768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7181.323975] LustreError: 277093:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7181.326969] LustreError: 277093:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7181.537937] Lustre: server umount lustre-MDT0000 complete [ 7184.356274] LustreError: 274495:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7184.384336] LustreError: 274495:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7186.056099] LustreError: 274480:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763408775 with bad export cookie 9085209716581945588 [ 7186.060922] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7186.079233] LustreError: 274480:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7186.270889] LustreError: 277295:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7186.354426] LustreError: 277295:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7186.710449] Lustre: server umount lustre-MDT0001 complete [ 7197.454593] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 7200.485328] Key type lgssc unregistered [ 7200.815784] LNet: 277827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7200.829489] LNetError: 277827:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7200.859520] LNet: Removed LNI 192.168.203.130@tcp [ 7201.720092] Key type .llcrypt unregistered [ 7201.729467] Key type ._llcrypt unregistered [ 7228.455977] Key type ._llcrypt registered [ 7228.457735] Key type .llcrypt registered [ 7228.562611] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 7229.433188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7229.478034] alg: No test for adler32 (adler32-zlib) [ 7230.526928] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 7230.746148] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 7232.447173] Key type lgssc registered [ 7233.886286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7242.259884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7242.287899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7244.030056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7248.110600] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7253.474048] LustreError: 279228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7253.493944] LustreError: 279228:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7256.246069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7256.838111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7261.229759] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7266.652118] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7270.403407] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7278.159471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7278.653803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7286.053934] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7288.918430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7292.123310] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7317.474545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7317.485050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7317.502084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7319.520967] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7319.523559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7319.524138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7319.538414] Lustre: Skipped 1 previous similar message [ 7320.633100] LustreError: 282250:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7320.736928] Lustre: server umount lustre-OST0000 complete [ 7324.642408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7324.656663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7329.761000] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7329.764648] Lustre: Skipped 2 previous similar messages [ 7330.595777] LustreError: 282451:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7330.598206] LustreError: 282451:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7330.785196] Lustre: server umount lustre-MDT0000 complete [ 7334.364599] LustreError: 281441:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763408924 with bad export cookie 8400878835787604305 [ 7334.368495] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7334.397598] LustreError: 281441:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7334.569093] LustreError: 282653:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7334.583459] LustreError: 282653:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7334.900574] Lustre: server umount lustre-MDT0001 complete [ 7345.482196] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 7348.560748] Key type lgssc unregistered [ 7348.864393] LNet: 283185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7348.875212] LNetError: 283185:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7348.890842] LNet: Removed LNI 192.168.203.130@tcp [ 7349.875868] Key type .llcrypt unregistered [ 7349.878450] Key type ._llcrypt unregistered [ 7368.149948] Key type ._llcrypt registered [ 7368.151542] Key type .llcrypt registered [ 7368.294800] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 7369.288138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7369.401922] alg: No test for adler32 (adler32-zlib) [ 7370.509917] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 7370.752794] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 7372.464260] Key type lgssc registered [ 7373.449561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7380.493851] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7380.516270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7382.145692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7385.482518] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7391.721379] LustreError: 284416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7391.746717] LustreError: 284416:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7392.824259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7393.328116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7397.409467] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7402.462542] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7405.584739] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7411.722901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7412.219456] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7417.652931] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7421.811463] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7426.557655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7436.770599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7436.772702] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7436.775666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7436.778371] Lustre: Skipped 1 previous similar message [ 7441.861895] LustreError: 286820:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7441.923280] LustreError: 286026:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7441.945906] LustreError: 286026:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7442.039513] Lustre: server umount lustre-OST0000 complete [ 7447.016975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7447.052384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7447.066553] Lustre: Skipped 2 previous similar messages [ 7451.416314] LustreError: 287021:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7451.421588] LustreError: 287021:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7451.564656] Lustre: server umount lustre-MDT0000 complete [ 7452.134166] LustreError: 284417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7452.154595] LustreError: 284417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7455.050604] LustreError: 284369:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763409044 with bad export cookie 3854218323787132629 [ 7455.064414] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7455.079865] LustreError: 284369:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7455.276824] LustreError: 287223:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7455.279827] LustreError: 287223:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7455.527472] Lustre: server umount lustre-MDT0001 complete [ 7466.315116] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 7469.135165] Key type lgssc unregistered [ 7469.413546] LNet: 287756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7469.431359] LNetError: 287756:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7470.501242] LNet: Removed LNI 192.168.203.130@tcp [ 7471.185476] Key type .llcrypt unregistered [ 7471.187854] Key type ._llcrypt unregistered [ 7483.527616] Key type ._llcrypt registered [ 7483.540843] Key type .llcrypt registered [ 7483.645543] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7498.409199] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 7510.204858] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 7510.980786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7511.024129] alg: No test for adler32 (adler32-zlib) [ 7511.992967] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 7512.166494] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 7513.831190] Key type lgssc registered [ 7514.776549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7521.920165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7527.166915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7532.654510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7538.209270] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7544.365415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7544.416284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7544.447622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7545.707188] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7545.748064] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7545.815347] Lustre: lustre-MDT0000: new disk, initializing [ 7545.874855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7545.892203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7549.587985] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7559.543675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7559.620252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7559.671739] Lustre: 291244:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7559.718108] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7559.720920] Lustre: Skipped 1 previous similar message [ 7559.786658] Lustre: lustre-MDT0001: new disk, initializing [ 7559.882720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7559.914696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7559.922474] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7563.383727] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7570.165893] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7573.943125] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7579.770714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7579.829179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7580.044608] Lustre: lustre-OST0000: new disk, initializing [ 7580.050693] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7580.119117] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7584.767994] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7585.827830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7585.834585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7585.940317] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7591.294989] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7595.609911] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7595.829705] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7600.306715] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7600.603087] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7606.244758] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7606.249887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7606.259495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7606.754103] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7606.783481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7609.030613] LustreError: 294183:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7609.416521] Lustre: server umount lustre-OST0000 complete [ 7616.488464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7616.491252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7616.506786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7619.590939] LustreError: 294385:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7619.598161] LustreError: 294385:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7619.850920] Lustre: server umount lustre-MDT0000 complete [ 7623.051477] LustreError: 292988:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763409212 with bad export cookie 11860053989720098614 [ 7623.063515] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7623.083520] LustreError: 292988:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7623.224480] LustreError: 294586:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7623.231452] LustreError: 294586:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7623.422464] Lustre: server umount lustre-MDT0001 complete [ 7630.729502] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 14:53:39 (1763409219) [ 7635.947337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7636.531539] LustreError: 295257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7636.540728] LustreError: 295257:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7636.598935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7640.274784] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7642.084506] LustreError: 295257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7647.208405] LustreError: 295256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7647.973409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7648.353169] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7651.840579] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7656.919883] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7660.481695] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7666.447325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7666.822564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7671.010232] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7675.912499] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7693.799107] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7693.808283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7693.819235] Lustre: Skipped 1 previous similar message [ 7693.825610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7693.829601] Lustre: Skipped 1 previous similar message [ 7697.889493] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7697.892847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7697.904265] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7699.460536] LustreError: 297525:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7699.462562] LustreError: 297525:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7699.578448] Lustre: server umount lustre-OST0000 complete [ 7708.128793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7708.136913] Lustre: Skipped 4 previous similar messages [ 7709.131381] LustreError: 297728:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7709.140850] LustreError: 297728:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7709.302513] Lustre: server umount lustre-MDT0000 complete [ 7712.955873] LustreError: 295236:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763409302 with bad export cookie 11860053989720099496 [ 7712.966164] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7712.976371] LustreError: 295236:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7713.272488] Lustre: server umount lustre-MDT0001 complete [ 7721.263754] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 7723.901121] Key type lgssc unregistered [ 7724.298538] LNet: 298462:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7724.306623] LNetError: 298462:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7724.335741] LNet: Removed LNI 192.168.203.130@tcp [ 7725.021321] Key type .llcrypt unregistered [ 7725.024392] Key type ._llcrypt unregistered [ 7748.357113] Key type ._llcrypt registered [ 7748.358506] Key type .llcrypt registered [ 7748.448824] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 7761.898929] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 7762.865600] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7763.001043] alg: No test for adler32 (adler32-zlib) [ 7764.254356] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 7764.503188] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 7766.167171] Key type lgssc registered [ 7767.198122] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7775.745783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7782.216277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7788.282702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7794.060923] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7798.789967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7803.643103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7810.232372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7810.276081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7810.285624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7811.516516] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7811.552352] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7811.663618] Lustre: lustre-MDT0000: new disk, initializing [ 7811.756861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7811.778215] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7814.775795] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7823.607914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7823.665794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7823.722407] Lustre: 302268:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7823.774729] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7823.778223] Lustre: Skipped 1 previous similar message [ 7823.835251] Lustre: lustre-MDT0001: new disk, initializing [ 7823.914267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7823.975687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7823.980120] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7827.609165] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7834.121715] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7837.334946] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7843.804960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7843.916797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7844.256404] Lustre: lustre-OST2710: new disk, initializing [ 7844.262057] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 7844.350225] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7850.082459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7850.095940] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7850.247916] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 7850.592657] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7858.159419] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 7864.410175] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7864.476527] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7864.599482] Lustre: lustre-OST03e8: new disk, initializing [ 7864.609217] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 7864.683533] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7869.722350] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7870.040905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7870.053935] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7870.133646] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 7874.324092] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 7879.457499] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 7879.627382] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7883.019378] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 7883.286895] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7887.332386] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 7887.550214] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7891.046897] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 7891.233613] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7895.526502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7895.537400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7895.550632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7896.038448] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7896.064919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7896.071505] Lustre: Skipped 2 previous similar messages [ 7900.516633] LustreError: 306612:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7900.578980] Lustre: server umount lustre-MDT0000 complete [ 7903.426574] LustreError: 302261:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763409493 with bad export cookie 9231189058964621872 [ 7903.434136] LustreError: 302261:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7903.434485] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7903.602730] LustreError: 306814:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7903.605400] LustreError: 306814:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7903.841340] Lustre: server umount lustre-MDT0001 complete [ 7917.632592] LustreError: 307015:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7917.637396] LustreError: 307015:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7917.669739] Lustre: server umount lustre-OST2710 complete [ 7929.544635] LustreError: 307216:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7929.552166] LustreError: 307216:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7929.685310] Lustre: server umount lustre-OST03e8 complete [ 7944.254140] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 7951.380807] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 7958.237719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7962.440124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7966.401863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7970.919435] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7975.708191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7975.764391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7975.904783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7975.926756] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7976.002525] Lustre: lustre-MDT0000: new disk, initializing [ 7976.091870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7976.110144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7979.132336] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7987.500399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7987.553554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7987.631811] Lustre: 310014:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7987.685062] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7987.688140] Lustre: Skipped 1 previous similar message [ 7987.731983] Lustre: lustre-MDT0001: new disk, initializing [ 7987.833701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7987.868821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7987.878395] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7991.350102] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 7998.534948] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8001.366709] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8007.740217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8007.832185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8008.031724] Lustre: lustre-OST0000: new disk, initializing [ 8008.037118] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8008.123337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8012.902882] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8013.902533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8013.914468] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8013.940931] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8018.456489] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8021.328823] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8021.432891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8024.026685] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8024.180825] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8029.668399] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8029.670127] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8029.672303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8029.677679] LustreError: Skipped 1 previous similar message [ 8029.701100] Lustre: Skipped 3 previous similar messages [ 8031.547178] LustreError: 312955:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8031.551189] LustreError: 312955:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8031.653095] Lustre: server umount lustre-OST0000 complete [ 8034.785917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8034.802416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8034.805242] Lustre: Skipped 1 previous similar message [ 8039.904089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8039.907446] Lustre: Skipped 2 previous similar messages [ 8048.095130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8048.183073] LustreError: 313156:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8048.186317] LustreError: 313156:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8048.274224] Lustre: server umount lustre-MDT0000 complete [ 8050.145114] LustreError: 310025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8050.151923] LustreError: 310025:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 8050.677314] LustreError: 310004:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763409640 with bad export cookie 9231189058964623552 [ 8050.678598] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8050.682081] LustreError: 310004:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8050.916978] Lustre: server umount lustre-MDT0001 complete [ 8056.110804] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 15:00:45 (1763409645) [ 8057.119610] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8058.507951] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 15:00:47 (1763409647) [ 8060.388353] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 8065.000596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8067.632595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8070.827177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8074.561732] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8075.477028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8085.972243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8086.231546] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8086.254174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8086.368408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8089.384262] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8096.319030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8096.422318] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8096.444651] Lustre: 315500:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8099.778393] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8103.986081] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8106.853547] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8112.279992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8112.345275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8112.514225] Lustre: Denying initial registration attempt from nid 192.168.203.130@tcp, specified as failover [ 8112.522662] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8112.527511] LustreError: 317117:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 8112.532528] LustreError: 317117:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 8112.537217] LustreError: 317117:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 8112.612150] Lustre: server umount lustre-OST0000 complete [ 8112.614364] LustreError: 317117:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8118.240362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8118.243186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8118.260420] Lustre: Skipped 2 previous similar messages [ 8118.264214] Lustre: Skipped 2 previous similar messages [ 8124.164369] LustreError: 317485:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 8124.171902] LustreError: 317485:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8124.226216] Lustre: server umount lustre-MDT0000 complete [ 8127.478378] LustreError: 315490:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763409717 with bad export cookie 9231189058964624427 [ 8127.483639] LustreError: 315490:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8127.485271] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8127.761743] Lustre: server umount lustre-MDT0001 complete [ 8135.822728] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 8138.260151] Key type lgssc unregistered [ 8138.523529] LNet: 318219:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8138.533771] LNetError: 318219:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8139.560854] LNet: Removed LNI 192.168.203.130@tcp [ 8140.381221] Key type .llcrypt unregistered [ 8140.388408] Key type ._llcrypt unregistered [ 8149.763818] Key type ._llcrypt registered [ 8149.767512] Key type .llcrypt registered [ 8149.842459] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 8150.378297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8150.452828] alg: No test for adler32 (adler32-zlib) [ 8151.453228] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 8151.627839] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 8153.335170] Key type lgssc registered [ 8154.348565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8159.341116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8161.749378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8164.402494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8167.328061] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8168.156259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8180.891851] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 8188.105491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8188.122112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8189.297368] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8189.319943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8189.432123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8192.412788] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8199.766560] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8199.846700] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8199.868990] Lustre: 320953:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8200.072898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8202.778567] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8206.322273] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8208.879181] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8213.614478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8213.675272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8213.817551] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8213.904129] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 8213.997487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8218.182983] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8222.693917] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8227.302254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8227.304188] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8227.314128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8227.332764] Lustre: Skipped 1 previous similar message [ 8230.720100] LustreError: 323237:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 8230.838109] Lustre: server umount lustre-OST0000 complete [ 8236.001817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8236.006059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8236.013274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8236.015989] Lustre: Skipped 1 previous similar message [ 8237.542482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8240.410788] LustreError: 323438:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 8240.416111] LustreError: 323438:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8240.485699] Lustre: server umount lustre-MDT0000 complete [ 8243.540892] LustreError: 320944:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763409833 with bad export cookie 17763101544689895022 [ 8243.546166] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8243.552023] LustreError: 320944:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8243.692627] LustreError: 323640:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8243.696644] LustreError: 323640:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8243.886528] Lustre: server umount lustre-MDT0001 complete [ 8252.043287] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 8254.491355] Key type lgssc unregistered [ 8254.761811] LNet: 324170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8254.768387] LNetError: 324170:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8254.789039] LNet: Removed LNI 192.168.203.130@tcp [ 8255.565513] Key type .llcrypt unregistered [ 8255.570644] Key type ._llcrypt unregistered [ 8275.411414] Key type ._llcrypt registered [ 8275.412766] Key type .llcrypt registered [ 8275.525239] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 8276.219908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8276.273080] alg: No test for adler32 (adler32-zlib) [ 8277.323749] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 8277.603560] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 8279.295555] Key type lgssc registered [ 8280.335783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8287.724365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8287.740476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8289.186884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8293.325232] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8294.368977] LustreError: 325548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8294.375235] LustreError: 325548:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8298.467962] LustreError: 325548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8300.506752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8300.757056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8303.952617] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8308.089543] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8311.082855] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8316.634587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8316.909848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8322.832972] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8327.135747] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8343.196287] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.30@tcp (stopping) [ 8344.544634] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8344.556481] Lustre: Skipped 2 previous similar messages [ 8344.557242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8344.562960] Lustre: Skipped 2 previous similar messages [ 8348.300993] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.30@tcp (stopping) [ 8349.340220] LustreError: 327817:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8349.422547] Lustre: server umount lustre-MDT0000 complete [ 8349.666778] LustreError: 327660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8349.686937] LustreError: 327660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8352.234599] LustreError: 325530:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763409941 with bad export cookie 10680076850501919082 [ 8352.240165] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8353.417760] LustreError: 327660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8353.450070] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.30@tcp (stopping) [ 8354.790859] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8357.792181] LustreError: 325545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8357.792670] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8357.801318] Lustre: Skipped 1 previous similar message [ 8357.801595] LustreError: 325545:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8358.437386] LustreError: 328021:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8358.447670] LustreError: 328021:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8358.611312] Lustre: server umount lustre-MDT0001 complete [ 8363.632593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8369.558515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8377.317041] LustreError: 324857:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000de0f4353 x1849069264729344/t0(0) o250->MGC192.168.203.130@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8377.755932] LustreError: 327158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8377.877213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8377.953359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8382.303025] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8388.074981] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8390.561197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8391.107217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8394.392683] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8396.269332] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8399.125914] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8400.594624] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8403.280076] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8404.496983] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8408.548337] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8408.551468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8408.564369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8411.620304] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8411.623844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8414.343173] LustreError: 330325:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8414.358195] LustreError: 330325:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8414.507906] Lustre: server umount lustre-OST0000 complete [ 8421.858607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8421.862178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8421.873886] Lustre: Skipped 1 previous similar message [ 8423.712662] LustreError: 330526:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8423.717878] LustreError: 330526:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8423.832216] Lustre: server umount lustre-MDT0000 complete [ 8426.661501] LustreError: 325528:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763410016 with bad export cookie 10680076850501926250 [ 8426.670160] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8426.674545] LustreError: 325528:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8426.970606] Lustre: server umount lustre-MDT0001 complete [ 8434.230993] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 8436.575865] Key type lgssc unregistered [ 8436.813413] LNet: 331261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8436.817321] LNetError: 331261:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8436.827505] LNet: Removed LNI 192.168.203.130@tcp [ 8437.374747] Key type .llcrypt unregistered [ 8437.376493] Key type ._llcrypt unregistered [ 8455.656232] Key type ._llcrypt registered [ 8455.658655] Key type .llcrypt registered [ 8455.740904] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 8456.549906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8456.562671] alg: No test for adler32 (adler32-zlib) [ 8457.516386] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 8457.665255] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 8459.319234] Key type lgssc registered [ 8460.205383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8467.118416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8467.141828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8468.542422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8471.566588] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8478.694568] LustreError: 332638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8478.702294] LustreError: 332638:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8479.189899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8479.658808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8483.399142] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8489.144190] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8492.860298] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8499.170654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8499.175157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8499.182954] Lustre: Skipped 1 previous similar message [ 8499.191050] Lustre: Skipped 1 previous similar message [ 8502.189113] LustreError: 334028:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8502.290650] Lustre: server umount lustre-MDT0000 complete [ 8504.288163] LustreError: 332633:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8504.307289] LustreError: 332633:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8505.315059] LustreError: 332638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8505.350425] LustreError: 332638:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8506.229827] LustreError: 332619:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763410095 with bad export cookie 6981335600385349107 [ 8506.236674] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8506.239455] LustreError: 332619:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8506.405601] LustreError: 334229:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8506.416857] LustreError: 334229:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8506.563846] Lustre: server umount lustre-MDT0001 complete [ 8517.113218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8517.342731] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8517.367145] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8517.557272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8520.588184] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8527.218236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8527.281626] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8527.309975] Lustre: 335001:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8527.527856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8530.461641] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8535.119545] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8538.234773] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8545.160043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8545.307083] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8545.314602] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8545.319928] LustreError: 336659:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 8545.325077] LustreError: 336659:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 8545.328764] LustreError: 336659:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 8545.381526] Lustre: server umount lustre-OST0000 complete [ 8545.383479] LustreError: 336659:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8552.146299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8552.297918] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8552.459873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8557.006784] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8562.481455] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8562.765645] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 8569.033848] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8569.128326] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8569.210598] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8569.307576] Lustre: lustre-OST0001: new disk, initializing [ 8569.312537] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8569.402982] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8571.719820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8571.731887] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8571.847421] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8573.797174] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8577.100867] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8582.118485] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8582.122107] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8582.134187] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8582.155142] Lustre: Skipped 1 previous similar message [ 8587.244261] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8587.259048] Lustre: Skipped 2 previous similar messages [ 8592.353227] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8592.365762] Lustre: Skipped 1 previous similar message [ 8593.379136] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8593.427986] LustreError: 338780:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 8593.449456] LustreError: 338780:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8593.539351] Lustre: server umount lustre-OST0001 complete [ 8597.476135] LustreError: 337126:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8597.478898] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8597.500379] LustreError: 337126:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8597.513312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8597.517486] Lustre: Skipped 1 previous similar message [ 8602.597900] LustreError: 337695:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8602.605163] LustreError: 337695:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8602.627195] LustreError: 338983:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8602.633325] LustreError: 338983:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8602.700713] Lustre: server umount lustre-OST0000 complete [ 8607.717184] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8607.721426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8607.733952] Lustre: Skipped 2 previous similar messages [ 8607.752604] Lustre: Skipped 2 previous similar messages [ 8611.855977] LustreError: 339184:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8611.874786] LustreError: 339184:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8611.955717] Lustre: server umount lustre-MDT0000 complete [ 8612.836625] LustreError: 335008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8612.870969] LustreError: 335008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8615.160634] LustreError: 334992:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763410204 with bad export cookie 6981335600385349653 [ 8615.163287] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8615.182612] LustreError: 334992:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8615.452536] Lustre: server umount lustre-MDT0001 complete [ 8623.648736] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 8625.648304] Key type lgssc unregistered [ 8625.899618] LNet: 339917:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8625.912293] LNetError: 339917:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8625.924568] LNet: Removed LNI 192.168.203.130@tcp [ 8626.721808] Key type .llcrypt unregistered [ 8626.723822] Key type ._llcrypt unregistered [ 8653.255809] Key type ._llcrypt registered [ 8653.258244] Key type .llcrypt registered [ 8653.338881] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 8665.260196] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 8665.961052] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8666.052613] alg: No test for adler32 (adler32-zlib) [ 8667.069225] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 8667.246230] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 8668.903207] Key type lgssc registered [ 8670.082455] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8677.533713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8683.002435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8688.554102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8694.085587] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8699.935960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8700.021390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8700.031932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8701.341392] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8701.399996] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8701.531131] Lustre: lustre-MDT0000: new disk, initializing [ 8701.654955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8701.680669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8705.222593] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8714.384557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8714.464197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8714.541602] Lustre: 343847:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8714.584583] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8714.587388] Lustre: Skipped 1 previous similar message [ 8714.682610] Lustre: lustre-MDT0001: new disk, initializing [ 8714.843859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8714.905312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8714.911688] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8718.099556] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8724.978342] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8727.724520] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8733.314269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8733.399686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8733.656393] Lustre: lustre-OST0000: new disk, initializing [ 8733.667614] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8733.760833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8738.433699] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8740.428021] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8740.435100] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8740.525821] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8744.199503] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8747.596868] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8747.872758] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8751.346742] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8751.550428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8756.197505] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8756.200591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8756.205883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8756.211655] Lustre: Skipped 1 previous similar message [ 8759.686681] LustreError: 346786:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8759.829340] Lustre: server umount lustre-OST0000 complete [ 8765.921294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8765.928825] LustreError: Skipped 1 previous similar message [ 8765.935542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8765.960530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8765.963281] Lustre: Skipped 1 previous similar message [ 8769.407047] LustreError: 346987:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8769.418653] LustreError: 346987:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8769.581871] Lustre: server umount lustre-MDT0000 complete [ 8772.644074] LustreError: 344628:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763410362 with bad export cookie 13981040032781765518 [ 8772.653416] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8772.657181] LustreError: 344628:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8772.781226] LustreError: 347188:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8772.789418] LustreError: 347188:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8772.960463] Lustre: server umount lustre-MDT0001 complete [ 8780.639551] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 15:12:49 (1763410369) [ 8785.359414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8790.124037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8809.394713] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 8817.941342] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 8825.771917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8831.523502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8837.465573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8843.860530] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8850.942521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8851.030521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8851.292583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8851.362218] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8851.509688] Lustre: lustre-MDT0000: new disk, initializing [ 8851.652904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8851.701042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8856.457028] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8866.525884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8866.611717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8866.761160] Lustre: 350964:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8866.841663] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8866.844297] Lustre: Skipped 1 previous similar message [ 8866.898273] Lustre: lustre-MDT0001: new disk, initializing [ 8866.990997] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8867.034621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8867.045205] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8870.961458] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8877.805803] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8881.022606] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8887.124493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8887.192284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8887.350175] Lustre: lustre-OST0000: new disk, initializing [ 8887.353288] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8887.444949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8892.690305] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8895.552683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8895.564161] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8895.688981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8899.162263] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8902.998151] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8903.182501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8906.655816] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8906.893644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8911.330101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8911.334193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8911.338771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8911.338779] Lustre: Skipped 2 previous similar messages [ 8911.348033] LustreError: Skipped 1 previous similar message [ 8911.419744] Lustre: Skipped 2 previous similar messages [ 8915.004276] LustreError: 353900:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8915.009179] LustreError: 353900:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8915.152761] Lustre: server umount lustre-OST0000 complete [ 8918.496928] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8918.502417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8918.513360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8918.518796] Lustre: Skipped 1 previous similar message [ 8924.576062] LustreError: 354102:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8924.578987] LustreError: 354102:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8924.732513] Lustre: server umount lustre-MDT0000 complete [ 8926.696285] LustreError: 350972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8926.707109] LustreError: 350972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8928.064429] LustreError: 350955:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763410517 with bad export cookie 13981040032781766400 [ 8928.066580] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8928.108316] LustreError: 350955:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8928.382646] Lustre: server umount lustre-MDT0001 complete [ 8936.714328] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 15:15:25 (1763410525) [ 8944.214462] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 15:15:32 (1763410532) [ 8953.074574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8953.599419] LustreError: 355321:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8953.683106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8957.375202] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8958.949652] LustreError: 355321:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8964.064180] LustreError: 355321:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8965.774882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8969.781272] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8974.799541] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8977.959607] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8983.450892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8989.279171] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 8993.960760] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9006.947331] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9008.556447] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9010.059804] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9011.725671] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9013.229912] Lustre: Failing over lustre-MDT0000 [ 9013.367220] LustreError: 357780:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9013.369644] LustreError: 357780:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 9013.453338] Lustre: server umount lustre-MDT0000 complete [ 9014.752607] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9014.756137] LustreError: 357482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9014.763117] Lustre: Skipped 2 previous similar messages [ 9024.993983] LustreError: 356499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9025.023133] LustreError: 356499:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 9031.148964] Lustre: 342062:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763410604/real 1763410604] req@000000007690dcb9 x1849069672697088/t0(0) o400->MGC192.168.203.130@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763410620 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9031.179698] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9032.757339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9040.543518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9040.549247] Lustre: Skipped 2 previous similar messages [ 9040.594601] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9042.058925] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9043.836878] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9045.999422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9046.035905] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9046.089121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9050.881252] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9052.389422] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9055.568982] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9056.980342] Lustre: Failing over lustre-MDT0000 [ 9057.125735] LustreError: 358862:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9057.127815] LustreError: 358862:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9057.262678] Lustre: server umount lustre-MDT0000 complete [ 9057.437514] LustreError: 356499:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9057.460050] LustreError: 356499:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 9061.347556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9061.347923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9061.365900] Lustre: Skipped 3 previous similar messages [ 9075.330439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9075.408766] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9075.653267] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9077.906666] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9079.200368] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9080.807880] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9080.810970] Lustre: Skipped 2 previous similar messages [ 9080.833545] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 9080.900296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9086.995885] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9088.504268] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9096.162404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9096.180285] Lustre: Skipped 2 previous similar messages [ 9096.183823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9096.186037] Lustre: Skipped 2 previous similar messages [ 9099.667530] LustreError: 359896:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9099.680812] LustreError: 359896:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9099.920197] Lustre: server umount lustre-MDT0000 complete [ 9101.285789] LustreError: 355321:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9101.306263] LustreError: 355321:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 9103.681065] LustreError: 355301:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763410693 with bad export cookie 13981040032781769095 [ 9103.683170] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9103.710612] LustreError: 355301:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9104.125778] Lustre: server umount lustre-MDT0001 complete [ 9118.225200] Lustre: server umount lustre-OST0000 complete [ 9133.101251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9133.528608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9133.532755] Lustre: Skipped 1 previous similar message [ 9137.444338] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9145.762426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9149.903611] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9155.587191] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9159.686447] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9166.199134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9171.939743] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9176.698206] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9178.808346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9186.275630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9186.284062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9186.300877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9191.501076] LustreError: 363371:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9191.506409] LustreError: 363371:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 9191.688481] Lustre: server umount lustre-OST0000 complete [ 9204.714292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9204.718880] Lustre: Skipped 6 previous similar messages [ 9209.320521] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9209.475993] Lustre: server umount lustre-MDT0000 complete [ 9209.824625] LustreError: 361838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9209.847810] LustreError: 361838:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9212.648695] LustreError: 361108:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763410802 with bad export cookie 13981040032781769830 [ 9212.654040] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9212.654344] LustreError: 361108:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9212.999355] Lustre: server umount lustre-MDT0001 complete [ 9223.258438] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 9226.197673] Key type lgssc unregistered [ 9226.572182] LNet: 364307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9226.587756] LNetError: 364307:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9226.608366] LNet: Removed LNI 192.168.203.130@tcp [ 9227.543753] Key type .llcrypt unregistered [ 9227.545633] Key type ._llcrypt unregistered [ 9252.895062] Key type ._llcrypt registered [ 9252.896504] Key type .llcrypt registered [ 9253.026455] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 9254.472981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9254.544510] alg: No test for adler32 (adler32-zlib) [ 9255.603198] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 9255.844365] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 9257.503191] Key type lgssc registered [ 9258.664983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9265.991655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9266.020280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9267.530725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9271.337587] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9277.924236] LustreError: 365806:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9277.947547] LustreError: 365806:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9278.820754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9279.329587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9282.578994] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9287.094333] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9292.076138] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9299.263256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9299.884658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9306.699927] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9311.672637] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9313.204884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9323.490510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9323.494735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9323.550480] Lustre: Skipped 2 previous similar messages [ 9323.576916] Lustre: Skipped 1 previous similar message [ 9327.715937] LustreError: 368055:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9327.867907] Lustre: server umount lustre-MDT0000 complete [ 9328.608884] LustreError: 366872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9328.633687] LustreError: 366872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9331.139167] LustreError: 365789:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763410920 with bad export cookie 4168495621780517848 [ 9331.140850] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9331.170345] LustreError: 365789:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9331.413896] LustreError: 368256:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9331.422486] LustreError: 368256:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9331.674835] Lustre: server umount lustre-MDT0001 complete [ 9345.693953] LustreError: 368458:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9345.697137] LustreError: 368458:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9345.875952] Lustre: server umount lustre-OST0000 complete [ 9359.501540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9360.262880] LustreError: 369189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9360.358103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9364.864987] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9365.473364] LustreError: 369189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9369.570476] LustreError: 369189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9373.729306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9374.351557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9378.786745] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9384.906321] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9388.802840] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9396.151269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9396.806589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9403.632137] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9404.984483] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9409.352322] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9425.377585] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9425.378627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9425.397679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9427.324228] LustreError: 371540:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9427.327409] LustreError: 371540:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9427.479769] Lustre: server umount lustre-MDT0000 complete [ 9430.502579] LustreError: 371349:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9430.542868] LustreError: 371349:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9431.716613] LustreError: 369169:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763411021 with bad export cookie 4168495621780518961 [ 9431.726549] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9431.732782] LustreError: 369169:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9431.841169] LustreError: 371741:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9431.844407] LustreError: 371741:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9431.990249] Lustre: server umount lustre-MDT0001 complete [ 9446.355771] LustreError: 371942:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9446.360106] LustreError: 371942:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9446.524309] Lustre: server umount lustre-OST0000 complete [ 9463.371470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9464.054084] LustreError: 372772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9464.134790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9468.405971] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9477.573856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9482.164588] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9487.296534] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9490.133463] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9495.945209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9496.174556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9496.179679] Lustre: Skipped 1 previous similar message [ 9501.362963] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9505.774238] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9508.475926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9519.081894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9519.082771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9519.098230] Lustre: Skipped 2 previous similar messages [ 9519.105518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9519.112676] LustreError: Skipped 1 previous similar message [ 9519.114610] Lustre: Skipped 2 previous similar messages [ 9520.719126] LustreError: 375017:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9520.731868] LustreError: 375017:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9520.941153] Lustre: server umount lustre-OST0000 complete [ 9529.315530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9529.317601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9529.324443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9529.342217] Lustre: Skipped 2 previous similar messages [ 9529.345156] Lustre: Skipped 1 previous similar message [ 9530.807549] Lustre: server umount lustre-MDT0000 complete [ 9534.293577] LustreError: 372753:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763411123 with bad export cookie 4168495621780519983 [ 9534.297845] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9534.320547] LustreError: 372753:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9534.865378] Lustre: server umount lustre-MDT0001 complete [ 9546.376683] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 9548.977974] Key type lgssc unregistered [ 9549.247073] LNet: 375953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9549.256266] LNetError: 375953:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9549.276303] LNet: Removed LNI 192.168.203.130@tcp [ 9550.219425] Key type .llcrypt unregistered [ 9550.228467] Key type ._llcrypt unregistered [ 9573.002879] Key type ._llcrypt registered [ 9573.004719] Key type .llcrypt registered [ 9573.109419] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 9574.361929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9574.368450] alg: No test for adler32 (adler32-zlib) [ 9575.401594] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 9575.650033] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 9577.335199] Key type lgssc registered [ 9578.461957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9585.528741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9585.553222] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9585.559131] LustreError: 377374:0:(osd_handler.c:8577:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9585.572492] LustreError: 377374:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9585.576266] LustreError: 377374:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9585.579971] LustreError: 377374:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9585.584110] LustreError: 377374:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9592.845218] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9592.855395] LustreError: 377772:0:(osd_handler.c:8577:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9592.883497] LustreError: 377772:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9592.901537] LustreError: 377772:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9592.910657] LustreError: 377772:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9592.924690] LustreError: 377772:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9611.140764] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 9613.946415] Key type lgssc unregistered [ 9614.205418] LNet: 378758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9614.217984] LNetError: 378758:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9614.235518] LNet: Removed LNI 192.168.203.130@tcp [ 9615.045706] Key type .llcrypt unregistered [ 9615.047530] Key type ._llcrypt unregistered [ 9631.764725] Key type ._llcrypt registered [ 9631.766304] Key type .llcrypt registered [ 9631.870863] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 9646.122609] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 9647.442138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9647.458820] alg: No test for adler32 (adler32-zlib) [ 9648.822496] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 9649.148492] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 9650.897860] Key type lgssc registered [ 9652.213619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9661.407084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9669.583961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9676.081217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9682.470346] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9689.749600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9689.793207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9689.804369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9691.195261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9691.248366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9691.362401] Lustre: lustre-MDT0000: new disk, initializing [ 9691.467923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9691.499727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9695.528113] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9705.630902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9705.717682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9705.834128] Lustre: 381890:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9705.939626] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9705.943126] Lustre: Skipped 1 previous similar message [ 9706.012816] Lustre: lustre-MDT0001: new disk, initializing [ 9706.164567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9706.240667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9706.251102] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9710.631414] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9717.971356] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9721.765981] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9729.400330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9729.500347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9729.784988] Lustre: lustre-OST0000: new disk, initializing [ 9729.798804] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9729.926573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9735.436748] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9737.810807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9737.820279] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9737.955605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9742.729318] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9747.557289] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9747.766743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9752.631436] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9752.895406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9758.690208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9758.704468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9758.706387] LustreError: Skipped 1 previous similar message [ 9758.714819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9758.748113] Lustre: Skipped 1 previous similar message [ 9761.049102] LustreError: 384829:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9761.259599] Lustre: server umount lustre-OST0000 complete [ 9767.905622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9767.916988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9767.948156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9767.951627] Lustre: Skipped 1 previous similar message [ 9768.934502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9768.958752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9768.969894] Lustre: Skipped 1 previous similar message [ 9771.593450] LustreError: 385031:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9771.625950] LustreError: 385031:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9771.765949] Lustre: server umount lustre-MDT0000 complete [ 9775.885428] LustreError: 381884:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763411365 with bad export cookie 10378814678091866787 [ 9775.924448] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9776.138835] LustreError: 385232:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9776.141777] LustreError: 385232:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9776.473134] Lustre: server umount lustre-MDT0001 complete [ 9787.364139] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9789.573225] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 15:29:38 (1763411378) [ 9796.822445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9797.447993] LustreError: 385953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9797.489574] LustreError: 385953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9797.573538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9802.174744] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9802.769463] LustreError: 385953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9807.850204] LustreError: 385953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9810.761532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9811.241173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9815.571901] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9820.555687] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9824.116185] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9831.044130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9831.516041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9837.247822] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9842.296912] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9848.617431] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9848.698571] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9848.803476] Lustre: lustre-OST0001: new disk, initializing [ 9848.815924] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9848.895444] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9854.513868] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [ 9858.122314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9858.131422] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9858.242487] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9861.272775] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9868.466702] Lustre: lustre-OST0001: Not available for connect from 192.168.203.30@tcp (stopping) [ 9868.774215] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9868.786315] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9868.812349] LustreError: Skipped 1 previous similar message [ 9868.842461] Lustre: Skipped 1 previous similar message [ 9871.100188] LustreError: 389154:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 9871.127222] LustreError: 389154:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9871.254838] Lustre: server umount lustre-OST0001 complete [ 9873.548765] LustreError: 387550:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.30@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9873.562915] LustreError: 387550:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9879.020633] LustreError: 387550:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9879.054207] LustreError: 387550:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9889.249298] LustreError: 388964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9889.252527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9889.271179] LustreError: 388964:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9889.282047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9889.282055] Lustre: Skipped 2 previous similar messages [ 9899.490170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9899.513617] Lustre: Skipped 4 previous similar messages [ 9902.047134] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9902.089158] LustreError: 389380:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9902.114300] LustreError: 389380:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9902.268899] Lustre: server umount lustre-OST0000 complete [ 9908.707233] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9908.726506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9908.763662] Lustre: Skipped 1 previous similar message [ 9912.649899] LustreError: 389584:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 9912.669752] LustreError: 389584:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9912.842070] Lustre: server umount lustre-MDT0000 complete [ 9914.850161] LustreError: 385953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9914.879736] LustreError: 385953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9916.447562] LustreError: 385933:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763411506 with bad export cookie 10378814678091867669 [ 9916.457186] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9916.469184] LustreError: 385933:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9923.047509] Lustre: server umount lustre-MDT0001 complete [ 9931.656769] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [ 9934.210994] Key type lgssc unregistered [ 9934.495496] LNet: 390317:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9934.506803] LNetError: 390317:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9934.531722] LNet: Removed LNI 192.168.203.130@tcp [ 9935.311166] Key type .llcrypt unregistered [ 9935.313430] Key type ._llcrypt unregistered [ 9964.073614] Key type ._llcrypt registered [ 9964.075295] Key type .llcrypt registered [ 9964.167126] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [ 9977.368646] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [ 9978.198562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9978.207872] alg: No test for adler32 (adler32-zlib) [ 9979.206939] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 9979.405720] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [ 9981.055186] Key type lgssc registered [ 9982.110789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9989.958055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9994.769159] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10000.127199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10006.292633] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10012.783263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10012.842870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10012.868452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10014.105463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10014.130871] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10014.213933] Lustre: lustre-MDT0000: new disk, initializing [10014.300631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10014.320544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10017.836215] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10027.108338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10027.160169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10027.230586] Lustre: 394268:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10027.284140] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10027.288256] Lustre: Skipped 1 previous similar message [10027.376730] Lustre: lustre-MDT0001: new disk, initializing [10027.455243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10027.488777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10027.504994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10030.945494] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10037.911873] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10041.373368] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10047.443417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10047.531830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10047.819444] Lustre: lustre-OST0000: new disk, initializing [10047.832142] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10047.966481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10053.240850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10053.263657] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10053.426944] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10054.665935] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10062.637438] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10066.153394] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10066.402231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10070.287988] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10070.521232] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10074.080546] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10074.085743] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10074.094650] LustreError: Skipped 1 previous similar message [10074.095479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10074.114845] Lustre: Skipped 1 previous similar message [10079.205319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10079.209070] Lustre: Skipped 2 previous similar messages [10084.329537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10084.332451] Lustre: Skipped 1 previous similar message [10086.367159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10086.414108] LustreError: 397206:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10086.509314] Lustre: server umount lustre-OST0000 complete [10094.058616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10094.062282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10094.094345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10096.414783] LustreError: 397408:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10096.423974] LustreError: 397408:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10096.509274] Lustre: server umount lustre-MDT0000 complete [10099.805138] LustreError: 394258:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763411689 with bad export cookie 10131237465697133405 [10099.814849] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10099.816643] LustreError: 394258:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10099.952599] LustreError: 397611:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10099.956920] LustreError: 397611:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10100.170120] Lustre: server umount lustre-MDT0001 complete [10108.420032] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 15:34:57 (1763411697) [10116.566391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10123.995477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10124.331357] Lustre: lustre-MDT0000: new disk, initializing [10124.371862] LustreError: 398879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10124.383521] LustreError: 398879:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10124.439776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10128.463689] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10129.901955] LustreError: 398879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10135.021770] LustreError: 398878:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10137.005391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10137.308937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10141.390903] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10146.909497] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10150.167182] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10152.932097] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10152.939156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10152.945390] Lustre: Skipped 1 previous similar message [10152.948239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10152.951144] Lustre: Skipped 1 previous similar message [10154.977644] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10158.177146] LustreError: 400221:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10158.182153] LustreError: 400221:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10158.283904] Lustre: server umount lustre-MDT0000 complete [10160.098320] LustreError: 398874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10160.122811] LustreError: 398874:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10161.485379] LustreError: 400222:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763411751 with bad export cookie 10131237465697134280 [10161.496659] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10161.816939] Lustre: server umount lustre-MDT0001 complete [10169.634493] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 15:35:58 (1763411758) [10175.865148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10176.279127] LustreError: 401143:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10176.362511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10180.418913] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10185.697165] LustreError: 401142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10185.736101] LustreError: 401142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10188.246853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10192.182170] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10197.051318] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10200.258607] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10206.719546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10207.060980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10207.064221] Lustre: Skipped 1 previous similar message [10212.454121] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10217.262974] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10229.413865] Lustre: Permanently deactivating lustre-OST0000 [10229.430322] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10235.929551] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10237.443829] LustreError: 403741:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10237.473737] LustreError: 403741:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [10237.485138] LustreError: 403741:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10240.183131] LustreError: 403789:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [10240.197723] LustreError: 403789:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10248.297238] LustreError: 403886:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10248.300046] LustreError: 403886:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [10248.461837] Lustre: server umount lustre-OST0000 complete [10251.216177] LustreError: 404039:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [10251.225357] LustreError: 404039:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10255.327874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10255.335711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10255.357656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10255.360620] Lustre: Skipped 2 previous similar messages [10262.510398] LustreError: 404136:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10262.514880] LustreError: 404136:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10262.586792] Lustre: server umount lustre-MDT0000 complete [10266.118750] LustreError: 401124:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763411855 with bad export cookie 10131237465697134840 [10266.123742] LustreError: 401124:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10266.127708] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10276.565626] Lustre: server umount lustre-MDT0001 complete [10284.989565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10288.644505] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10290.862860] LustreError: 405182:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10297.321825] Lustre: 405521:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.130@tcp [10299.156899] LustreError: 405618:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [10299.159599] LustreError: 405618:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [10299.242459] Lustre: server umount MGS complete [10310.283168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10310.605093] LustreError: 406248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10310.620128] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10310.628528] Lustre: Skipped 1 previous similar message [10310.688455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10314.640466] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10322.119238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10322.431736] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10326.152980] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10331.480287] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10335.554419] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10337.765554] Lustre: Permanently reactivating lustre-OST0000 [10337.802303] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10337.805573] Lustre: Skipped 2 previous similar messages [10363.871160] LustreError: 407758:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10371.520675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10376.644404] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10381.530297] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10389.480973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10389.488890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10389.503861] Lustre: Skipped 1 previous similar message [10389.504805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10389.504809] Lustre: Skipped 4 previous similar messages [10389.543752] LustreError: Skipped 1 previous similar message [10391.753814] LustreError: 409367:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10391.756957] LustreError: 409367:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [10391.928324] Lustre: server umount lustre-OST0000 complete [10399.200264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10401.841527] Lustre: server umount lustre-MDT0000 complete [10404.835458] LustreError: 408332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10404.846526] LustreError: 408332:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10405.245497] LustreError: 406229:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763411994 with bad export cookie 10131237465697135792 [10405.255309] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10405.259592] LustreError: 406229:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [10405.538519] Lustre: server umount lustre-MDT0001 complete [10414.078786] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [10416.810552] Key type lgssc unregistered [10417.133076] LNet: 410306:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10417.148472] LNetError: 410306:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10418.224537] LNet: Removed LNI 192.168.203.130@tcp [10419.017684] Key type .llcrypt unregistered [10419.019419] Key type ._llcrypt unregistered [10434.025537] Key type ._llcrypt registered [10434.027036] Key type .llcrypt registered [10434.133405] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [10446.164880] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [10447.016421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10447.168171] alg: No test for adler32 (adler32-zlib) [10448.219237] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [10448.410189] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [10450.095191] Key type lgssc registered [10451.478992] Lustre: Echo OBD driver; http://www.lustre.org/ [10460.624975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10466.173463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10471.094326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10477.370138] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10483.455061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10483.508210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10483.569531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10484.745182] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10484.778777] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10484.837471] Lustre: lustre-MDT0000: new disk, initializing [10484.944164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10484.972415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10489.225656] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10499.513695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10499.596773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10499.670277] Lustre: 413416:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10499.729446] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10499.732673] Lustre: Skipped 1 previous similar message [10499.785123] Lustre: lustre-MDT0001: new disk, initializing [10499.915460] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10499.967418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10499.985940] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10503.642254] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10511.544286] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10515.315852] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10522.283137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10522.385085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10522.884170] Lustre: lustre-OST0000: new disk, initializing [10522.893981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10522.987807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10528.405684] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10529.376226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10529.380501] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10529.496253] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10535.693274] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10540.320836] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10540.516678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10544.335589] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10544.492125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10548.194165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10548.202813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10552.211323] LustreError: 416358:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10552.352374] Lustre: server umount lustre-OST0000 complete [10556.398891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10556.404498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10556.410756] Lustre: Skipped 1 previous similar message [10556.423444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10556.425660] Lustre: Skipped 1 previous similar message [10558.432707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10558.436599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10558.460645] Lustre: Skipped 1 previous similar message [10561.720831] LustreError: 416560:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10561.724938] LustreError: 416560:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10561.911079] Lustre: server umount lustre-MDT0000 complete [10565.281187] LustreError: 413408:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763412154 with bad export cookie 2001987154895820557 [10565.286647] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10565.345583] LustreError: 413408:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10565.500892] LustreError: 416761:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10565.514516] LustreError: 416761:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10565.741569] Lustre: server umount lustre-MDT0001 complete [10573.113307] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10574.617190] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 15:42:43 (1763412163) [10580.710337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10581.118836] LustreError: 417481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10581.130420] LustreError: 417481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10581.178142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10585.115814] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10586.593648] LustreError: 417481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10590.698524] LustreError: 417481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10593.787984] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10594.188535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10598.649808] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10603.830817] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10607.135544] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10613.475756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10613.852372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10618.947646] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10623.153603] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10625.004771] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10627.167395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10631.649150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10631.652444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10631.686931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10636.776930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10636.779820] Lustre: Skipped 2 previous similar messages [10637.122153] LustreError: 419920:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10637.134532] LustreError: 419920:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10637.291877] Lustre: server umount lustre-OST0000 complete [10641.890994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10641.915158] Lustre: Skipped 2 previous similar messages [10646.761031] LustreError: 420121:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10646.773170] LustreError: 420121:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10646.820964] Lustre: server umount lustre-MDT0000 complete [10647.019301] LustreError: 418458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10647.041481] LustreError: 418458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10650.065066] LustreError: 417461:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763412239 with bad export cookie 2001987154895821432 [10650.081529] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10650.412441] Lustre: server umount lustre-MDT0001 complete [10660.594335] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [10662.971542] Key type lgssc unregistered [10663.196983] LNet: 420856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10663.206856] LNetError: 420856:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10663.237563] LNet: Removed LNI 192.168.203.130@tcp [10664.000329] Key type .llcrypt unregistered [10664.002675] Key type ._llcrypt unregistered [10683.808191] Key type ._llcrypt registered [10683.809456] Key type .llcrypt registered [10683.922816] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [10684.158490] Key type .llcrypt unregistered [10684.160084] Key type ._llcrypt unregistered [10698.887636] Key type ._llcrypt registered [10698.889283] Key type .llcrypt registered [10698.978987] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [10700.136717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10700.154749] alg: No test for adler32 (adler32-zlib) [10701.277909] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [10701.464056] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [10703.127305] Key type lgssc registered [10704.161941] Lustre: Echo OBD driver; http://www.lustre.org/ [10710.832825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10710.871673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10712.420442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10716.306397] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10722.788109] LustreError: 423025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10722.808456] LustreError: 423025:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10725.505599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10725.726180] LustreError: 423025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10725.754053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10730.084297] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10730.976891] LustreError: 423025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10730.989946] LustreError: 423025:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10734.443184] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10736.099576] LustreError: 423021:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10736.107495] LustreError: 423021:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10738.180342] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10741.025914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10741.398219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10744.665811] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10756.582918] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10756.590077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10756.611661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10760.573410] LustreError: 425077:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [10760.732182] Lustre: server umount lustre-OST0000 complete [10766.819460] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10766.823207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10766.829736] Lustre: Skipped 1 previous similar message [10766.837135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10766.843396] Lustre: Skipped 2 previous similar messages [10770.232423] LustreError: 425278:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10770.255143] LustreError: 425278:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10770.402300] Lustre: server umount lustre-MDT0000 complete [10772.448651] LustreError: 423021:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10772.462419] LustreError: 423021:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10773.477277] LustreError: 424991:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763412363 with bad export cookie 9640202961732951027 [10773.479782] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10773.489040] LustreError: 424991:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10773.595389] LustreError: 425482:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10773.601660] LustreError: 425482:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10773.706099] Lustre: server umount lustre-MDT0001 complete [10781.893189] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [10783.942459] Key type lgssc unregistered [10784.181957] LNet: 426014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10784.188155] LNetError: 426014:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10784.204476] LNet: Removed LNI 192.168.203.130@tcp [10784.951686] Key type .llcrypt unregistered [10784.953243] Key type ._llcrypt unregistered [10803.819459] Key type ._llcrypt registered [10803.820964] Key type .llcrypt registered [10803.916111] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [10804.594640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10804.607152] alg: No test for adler32 (adler32-zlib) [10805.685297] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [10805.921137] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [10807.575174] Key type lgssc registered [10808.425647] Lustre: Echo OBD driver; http://www.lustre.org/ [10814.297501] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10814.316928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10829.857909] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10834.979222] LustreError: 427363:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.130@tcp: failed processing log, type 4: rc = -110 [10861.664060] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10866.278703] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10871.046844] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10876.266170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10876.632442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10876.682666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10879.613092] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10883.044372] LustreError: 428389:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10883.070535] LustreError: 428389:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10885.760740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10886.002548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10888.543592] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10892.346350] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10894.744790] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10901.474560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10901.475392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10901.490027] Lustre: Skipped 1 previous similar message [10906.603479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10906.611274] Lustre: Skipped 2 previous similar messages [10906.746687] LustreError: 429643:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [10906.871380] Lustre: server umount lustre-OST0000 complete [10911.714736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10911.715271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10911.724431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10911.725255] Lustre: Skipped 1 previous similar message [10915.616871] LustreError: 429846:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [10915.619496] LustreError: 429846:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10915.681134] Lustre: server umount lustre-MDT0000 complete [10916.833483] LustreError: 429820:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10916.841015] LustreError: 429820:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10918.573419] LustreError: 429013:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763412508 with bad export cookie 2065986273777361873 [10918.580092] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10918.580864] LustreError: 429013:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [10918.691345] LustreError: 430048:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10918.694240] LustreError: 430048:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10918.830880] Lustre: server umount lustre-MDT0001 complete [10925.901755] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [10927.823272] Key type lgssc unregistered [10928.069796] LNet: 430580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10928.077909] LNetError: 430580:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10928.100775] LNet: Removed LNI 192.168.203.130@tcp [10928.703629] Key type .llcrypt unregistered [10928.705340] Key type ._llcrypt unregistered [10947.296863] Key type ._llcrypt registered [10947.298112] Key type .llcrypt registered [10947.412616] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [10948.171558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10948.343199] alg: No test for adler32 (adler32-zlib) [10949.364493] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [10949.627403] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [10951.295166] Key type lgssc registered [10952.363473] Lustre: Echo OBD driver; http://www.lustre.org/ [10958.649860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10958.667904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10960.148047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10963.991222] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10969.572557] LustreError: 431956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10969.588228] LustreError: 431956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10971.175540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10971.487097] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10974.584072] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10979.012520] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10982.140816] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10987.260188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10987.568771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10992.206507] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [10996.754402] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11004.975778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11010.018035] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11010.021068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11010.034649] Lustre: Skipped 1 previous similar message [11012.758933] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.30@tcp (stopping) [11012.762833] Lustre: Skipped 2 previous similar messages [11015.137925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11015.140841] Lustre: Skipped 2 previous similar messages [11015.965816] LustreError: 434220:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11016.042219] Lustre: server umount lustre-MDT0000 complete [11018.003829] LustreError: 431935:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.30@tcp arrived at 1763412607 with bad export cookie 11945443693807951042 [11018.011857] LustreError: 431935:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [11019.428462] LustreError: 431935:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763412609 with bad export cookie 11945443693807950370 [11019.438569] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11019.453525] LustreError: 431935:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [11020.257172] LustreError: 433025:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11020.261492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11020.273483] LustreError: 433025:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11020.305923] Lustre: Skipped 1 previous similar message [11020.326074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11024.289164] LustreError: 431955:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11024.311836] LustreError: 431955:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11025.542195] LustreError: 434423:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11025.547277] LustreError: 434423:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11025.675336] Lustre: server umount lustre-OST0000 complete [11029.413268] LustreError: 431952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11029.423652] LustreError: 431952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11031.305742] LustreError: 434776:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [11031.308960] LustreError: 434776:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11031.562992] Lustre: server umount lustre-MDT0001 complete [11040.101249] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [11042.480772] Key type lgssc unregistered [11042.702824] LNet: 435308:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11042.710206] LNetError: 435308:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11042.721558] LNet: Removed LNI 192.168.203.130@tcp [11043.348166] Key type .llcrypt unregistered [11043.349895] Key type ._llcrypt unregistered [11062.302447] Key type ._llcrypt registered [11062.304091] Key type .llcrypt registered [11062.388433] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [11062.910394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11062.945886] alg: No test for adler32 (adler32-zlib) [11063.974476] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [11064.166922] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [11065.807177] Key type lgssc registered [11066.980649] Lustre: Echo OBD driver; http://www.lustre.org/ [11073.971075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11073.982826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11075.479954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11079.542412] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11085.793991] LustreError: 436685:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11085.818497] LustreError: 436685:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11086.885419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11087.166818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11089.985070] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11094.193908] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11097.132722] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11102.300130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11102.530644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11107.114263] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11111.144359] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11121.017993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11126.246698] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11126.247693] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11126.261483] Lustre: Skipped 1 previous similar message [11126.947667] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.30@tcp (stopping) [11126.953059] Lustre: Skipped 1 previous similar message [11130.001554] LustreError: 438951:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [11130.068053] Lustre: server umount lustre-MDT0001 complete [11131.370801] LustreError: 438675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11131.382399] LustreError: 438675:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11136.484363] LustreError: 436686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11136.487922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11136.489557] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11136.513969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11140.538283] LustreError: 439153:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11140.540405] LustreError: 439153:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11140.620943] Lustre: server umount lustre-OST0000 complete [11141.607695] LustreError: 436686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11144.072917] LustreError: 439354:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11144.079796] LustreError: 439354:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11144.240128] Lustre: server umount lustre-MDT0000 complete [11155.379725] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [11157.713176] Key type lgssc unregistered [11157.978163] LNet: 440038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11157.993741] LNetError: 440038:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11158.015043] LNet: Removed LNI 192.168.203.130@tcp [11158.780254] Key type .llcrypt unregistered [11158.787459] Key type ._llcrypt unregistered [11176.645476] Key type ._llcrypt registered [11176.647669] Key type .llcrypt registered [11176.738222] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [11187.537521] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [11188.248693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11188.313762] alg: No test for adler32 (adler32-zlib) [11189.251475] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [11189.388612] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [11190.999160] Key type lgssc registered [11192.082955] Lustre: Echo OBD driver; http://www.lustre.org/ [11200.960628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11207.494081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11213.687984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11219.629946] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11227.448663] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [11235.308961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11240.783653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11240.821512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11240.833475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11242.041137] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11242.080807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11242.153506] Lustre: lustre-MDT0000: new disk, initializing [11242.252844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11242.271458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11245.192537] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11252.622495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11258.544618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11258.605269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11258.761443] Lustre: lustre-OST0000: new disk, initializing [11258.764996] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11258.769591] Lustre: Skipped 1 previous similar message [11258.824434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11263.659715] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11264.061672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11264.073490] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11264.107597] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11270.884935] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11278.587367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11284.737610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11284.810552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11284.867675] Lustre: 443994:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11284.932060] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11284.990823] Lustre: lustre-MDT0001: new disk, initializing [11285.110338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11285.164095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11285.183220] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11288.923123] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11296.774991] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11297.011335] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11300.371608] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11300.557610] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11306.147260] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.30@tcp (stopping) [11307.493182] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11307.493575] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11307.494693] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11307.505975] Lustre: Skipped 1 previous similar message [11310.841873] LustreError: 447415:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11311.083579] Lustre: server umount lustre-MDT0001 complete [11317.731810] LustreError: 444006:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11317.746310] LustreError: 444006:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11322.855645] LustreError: 445009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11322.863512] LustreError: 445009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11327.970703] LustreError: 447416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11327.971124] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11327.990548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11327.994026] Lustre: Skipped 1 previous similar message [11333.088383] LustreError: 447416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11333.091380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11338.213453] LustreError: 447416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11338.213938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11342.304601] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11342.366356] LustreError: 447688:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [11342.379449] LustreError: 447688:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11342.531940] Lustre: server umount lustre-OST0000 complete [11360.225891] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11360.339139] LustreError: 447890:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [11360.350162] LustreError: 447890:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11360.639155] Lustre: server umount lustre-MDT0000 complete [11373.420691] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [11376.059331] Key type lgssc unregistered [11376.323695] LNet: 448579:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11376.343657] LNetError: 448579:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11376.381202] LNet: Removed LNI 192.168.203.130@tcp [11377.129850] Key type .llcrypt unregistered [11377.131883] Key type ._llcrypt unregistered [11410.167783] Key type ._llcrypt registered [11410.169687] Key type .llcrypt registered [11410.254904] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [11410.500399] Key type .llcrypt unregistered [11410.505108] Key type ._llcrypt unregistered [11424.378914] Key type ._llcrypt registered [11424.380479] Key type .llcrypt registered [11424.530532] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [11425.374350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11425.480951] alg: No test for adler32 (adler32-zlib) [11426.485285] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [11426.689260] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [11428.327167] Key type lgssc registered [11429.338684] Lustre: Echo OBD driver; http://www.lustre.org/ [11436.589884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11442.080768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11447.312344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11452.870526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11452.924918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11452.946625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11454.148991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11454.185557] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11454.252790] Lustre: lustre-MDT0000: new disk, initializing [11454.317394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11454.336536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11457.734858] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11467.458608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11467.534455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11467.609754] Lustre: 451941:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11467.673632] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11467.676546] Lustre: Skipped 1 previous similar message [11467.756355] Lustre: lustre-MDT0001: new disk, initializing [11467.848903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11467.891801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11467.900473] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11471.027740] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11477.779348] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11480.607830] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11486.325096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11486.382914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11486.570858] Lustre: lustre-OST0000: new disk, initializing [11486.577501] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11486.646264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11491.984530] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11495.480733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11495.488029] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11495.631963] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11498.306659] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11506.145252] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11506.150334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11506.150873] Lustre: Skipped 1 previous similar message [11506.176093] Lustre: Skipped 2 previous similar messages [11509.956172] LustreError: 454384:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11510.151955] Lustre: server umount lustre-MDT0000 complete [11513.157743] LustreError: 451935:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763413102 with bad export cookie 16517432730799373736 [11513.163803] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11513.170262] LustreError: 451935:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [11513.305839] LustreError: 454586:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11513.312121] LustreError: 454586:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11513.518035] Lustre: server umount lustre-MDT0001 complete [11525.470653] LustreError: 454786:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [11525.473992] LustreError: 454786:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11525.731995] Lustre: server umount lustre-OST0000 complete [11534.745603] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 15:58:43 (1763413123) [11535.781688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11543.006588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11543.366510] LustreError: 455616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11543.384637] LustreError: 455616:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11543.450739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11547.059303] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11548.639982] LustreError: 455616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11553.765396] LustreError: 455616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11554.981944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11555.353409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11558.567392] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11563.396512] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11566.895980] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11573.009942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11573.074976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11573.200317] Lustre: Found index 0 for lustre-OST0000, updating log [11573.377825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11578.294109] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11582.502791] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11588.723988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [11599.335360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11599.335929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11599.344663] Lustre: Skipped 1 previous similar message [11599.346086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11600.423725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11600.441363] Lustre: Skipped 2 previous similar messages [11602.360079] LustreError: 457893:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11602.363986] LustreError: 457893:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11602.597270] Lustre: server umount lustre-OST0000 complete [11604.997313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11611.617574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11611.617600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11611.630463] Lustre: Skipped 1 previous similar message [11611.634537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11611.640378] Lustre: Skipped 1 previous similar message [11612.743818] LustreError: 458149:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11612.852506] LustreError: 458149:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11612.959613] Lustre: server umount lustre-MDT0000 complete [11615.731610] LustreError: 455597:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763413205 with bad export cookie 16517432730799375129 [11615.733991] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11615.741232] LustreError: 455597:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [11615.953901] Lustre: server umount lustre-MDT0001 complete [11622.194467] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 16:00:11 (1763413211) [11623.020303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11623.764566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11624.537604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11625.303359] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11625.970696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11626.727760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11627.457660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11628.243620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11628.882905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11629.498287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11630.166652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11631.079494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11631.859970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11632.632907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11633.465637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11634.387437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11635.309601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11636.113721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11636.860951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11637.794876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11638.684793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11639.493799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11639.775683] LNet: 1 local NIs in recovery (showing 1): 192.168.203.130@tcp [11640.349987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11641.051655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11641.975862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11642.811888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11643.510146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11644.154839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11644.962852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11645.741189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11646.571893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11647.365072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11648.081678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11648.981537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11649.736413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11650.591477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11651.404924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11652.167512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11652.999975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11653.761531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11654.616948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11655.534264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11656.269886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11656.997809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11657.857172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11658.766532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11659.664235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11660.735198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11661.633504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11662.670247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11663.748785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11670.981980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11671.768432] LustreError: 461745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11671.837887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11675.481314] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11677.152904] LustreError: 461745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11683.613851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11687.571124] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11692.290512] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11695.291554] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11700.946939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11701.028501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11701.187981] Lustre: Found index 0 for lustre-OST0000, updating log [11705.867559] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11709.763714] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11711.520657] LNet: 1 local NIs in recovery (showing 1): 192.168.203.130@tcp [11715.044475] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11715.047696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11715.053040] Lustre: Skipped 1 previous similar message [11715.057830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11717.569902] LustreError: 464011:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11717.582156] LustreError: 464011:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [11717.723048] Lustre: server umount lustre-OST0000 complete [11721.634774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11721.641183] Lustre: Skipped 1 previous similar message [11723.682194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11723.686316] Lustre: Skipped 5 previous similar messages [11726.824068] Lustre: server umount lustre-MDT0000 complete [11727.778257] LustreError: 461738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11727.797254] LustreError: 461738:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11730.251919] LustreError: 461726:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763413319 with bad export cookie 16517432730799376207 [11730.258131] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11730.264150] LustreError: 461726:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11730.625885] Lustre: server umount lustre-MDT0001 complete [11733.193633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11748.643544] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [11756.141383] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [11765.459166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11771.539908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11777.213303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11782.959294] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11789.309237] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 16:02:58 (1763413378) [11802.183444] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [11804.746374] Key type lgssc unregistered [11805.012220] LNet: 467984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11805.020114] LNetError: 467984:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11847.013904] LNet: Removed LNI 192.168.203.130@tcp [11847.686618] Key type .llcrypt unregistered [11847.693822] Key type ._llcrypt unregistered [11860.643441] Key type ._llcrypt registered [11860.644907] Key type .llcrypt registered [11860.755131] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [11861.551153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11861.635966] alg: No test for adler32 (adler32-zlib) [11862.633825] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [11862.807471] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [11864.471297] Key type lgssc registered [11865.319178] Lustre: Echo OBD driver; http://www.lustre.org/ [11868.501334] LNet: Added LNI 192.168.203.130@tcp100 [8/256/0/180] [11869.149635] LNet: Added LNI 192.168.203.130@tcp101 [8/256/0/180] [11869.802896] LNet: Added LNI 192.168.203.130@tcp102 [8/256/0/180] [11870.460495] LNet: Added LNI 192.168.203.130@tcp103 [8/256/0/180] [11871.245499] LNet: Added LNI 192.168.203.130@tcp104 [8/256/0/180] [11871.916809] LNet: Added LNI 192.168.203.130@tcp105 [8/256/0/180] [11872.590799] LNet: Added LNI 192.168.203.130@tcp106 [8/256/0/180] [11873.212455] LNet: Added LNI 192.168.203.130@tcp107 [8/256/0/180] [11873.906381] LNet: Added LNI 192.168.203.130@tcp108 [8/256/0/180] [11874.569560] LNet: Added LNI 192.168.203.130@tcp109 [8/256/0/180] [11875.161258] LNet: Added LNI 192.168.203.130@tcp110 [8/256/0/180] [11875.916680] LNet: Added LNI 192.168.203.130@tcp111 [8/256/0/180] [11876.590487] LNet: Added LNI 192.168.203.130@tcp112 [8/256/0/180] [11877.337816] LNet: Added LNI 192.168.203.130@tcp113 [8/256/0/180] [11878.092835] LNet: Added LNI 192.168.203.130@tcp114 [8/256/0/180] [11878.811276] LNet: Added LNI 192.168.203.130@tcp115 [8/256/0/180] [11879.548518] LNet: Added LNI 192.168.203.130@tcp116 [8/256/0/180] [11880.219193] LNet: Added LNI 192.168.203.130@tcp117 [8/256/0/180] [11880.907797] LNet: Added LNI 192.168.203.130@tcp118 [8/256/0/180] [11881.778246] LNet: Added LNI 192.168.203.130@tcp119 [8/256/0/180] [11882.515276] LNet: Added LNI 192.168.203.130@tcp120 [8/256/0/180] [11883.333649] LNet: Added LNI 192.168.203.130@tcp121 [8/256/0/180] [11884.001668] LNet: Added LNI 192.168.203.130@tcp122 [8/256/0/180] [11884.681752] LNet: Added LNI 192.168.203.130@tcp123 [8/256/0/180] [11885.575150] LNet: Added LNI 192.168.203.130@tcp124 [8/256/0/180] [11886.440460] LNet: Added LNI 192.168.203.130@tcp125 [8/256/0/180] [11887.230265] LNet: Added LNI 192.168.203.130@tcp126 [8/256/0/180] [11887.958977] LNet: Added LNI 192.168.203.130@tcp127 [8/256/0/180] [11888.882205] LNet: Added LNI 192.168.203.130@tcp128 [8/256/0/180] [11889.650964] LNet: Added LNI 192.168.203.130@tcp129 [8/256/0/180] [11890.450440] LNet: Added LNI 192.168.203.130@tcp130 [8/256/0/180] [11891.400837] LNet: Added LNI 192.168.203.130@tcp131 [8/256/0/180] [11892.133095] LNet: Added LNI 192.168.203.130@tcp132 [8/256/0/180] [11893.051320] LNet: Added LNI 192.168.203.130@tcp133 [8/256/0/180] [11893.953461] LNet: Added LNI 192.168.203.130@tcp134 [8/256/0/180] [11894.811454] LNet: Added LNI 192.168.203.130@tcp135 [8/256/0/180] [11901.372749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11901.444281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11901.481518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11902.637956] Lustre: 470963:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.130@tcp: can't add NID 192.168.203.130@tcp130: rc = -75 [11902.828662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11902.854363] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11902.976230] Lustre: lustre-MDT0000: new disk, initializing [11903.095905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11903.119631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11903.719740] LNetError: 470984:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [11903.733462] LNetError: Protocol error connecting to 192.168.203.130@tcp112 at host 192.168.203.130:988: is it running a compatible version of Lustre? [11906.931306] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11916.398990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11916.455087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11916.525883] Lustre: 470982:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11916.593416] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11916.604513] Lustre: Skipped 1 previous similar message [11916.678818] Lustre: lustre-MDT0001: new disk, initializing [11916.702718] Lustre: 471767:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.130@tcp: can't add NID 192.168.203.130@tcp130: rc = -75 [11916.708220] Lustre: 471767:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11916.788887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11916.838267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11916.858420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11918.615847] Lustre: 471881:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.130@tcp: can't add NID 192.168.203.130@tcp130: rc = -75 [11918.627921] Lustre: 471881:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11920.312995] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11927.897210] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11931.063836] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11937.258822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11937.339981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11937.611679] Lustre: lustre-OST0000: new disk, initializing [11937.625925] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11937.698059] Lustre: 472739:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.130@tcp: can't add NID 192.168.203.130@tcp130: rc = -75 [11937.703543] Lustre: 472739:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [11937.758289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11939.623340] LNetError: 470987:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [11939.634687] LNetError: Protocol error connecting to 192.168.203.130@tcp112 at host 192.168.203.130:988: is it running a compatible version of Lustre? [11942.781432] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [11943.462935] Lustre: 473104:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.130@tcp: can't add NID 192.168.203.130@tcp130: rc = -75 [11943.483419] Lustre: 473104:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11943.553461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11943.560461] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11943.686159] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11949.535454] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11952.933872] LNetError: 470988:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [11952.949750] LNetError: Protocol error connecting to 192.168.203.130@tcp112 at host 192.168.203.130:988: is it running a compatible version of Lustre? [11959.269039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11959.270498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11959.284869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11959.300741] LustreError: Skipped 1 previous similar message [11959.306369] Lustre: Skipped 1 previous similar message [11962.116307] LustreError: 473572:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11962.208287] Lustre: server umount lustre-OST0000 complete [11965.217876] LNetError: 470986:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [11965.223741] LNetError: Protocol error connecting to 192.168.203.130@tcp112 at host 192.168.203.130:988: is it running a compatible version of Lustre? [11967.970340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11967.974144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11967.982392] Lustre: Skipped 1 previous similar message [11967.985154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11969.504318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11969.508064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11971.428941] LustreError: 473774:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11971.433332] LustreError: 473774:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11971.506609] Lustre: server umount lustre-MDT0000 complete [11974.307802] LustreError: 470972:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763413564 with bad export cookie 3885547925628185399 [11974.309548] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11974.315257] LustreError: 470972:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11974.461566] LustreError: 473976:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11974.464387] LustreError: 473976:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11974.627425] Lustre: server umount lustre-MDT0001 complete [11983.635900] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [11986.001387] Key type lgssc unregistered [11986.333566] LNet: 474514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11986.345175] LNetError: 474514:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11986.350556] LNet: Removed LNI 192.168.203.130@tcp [11987.359179] LNet: Removed LNI 192.168.203.130@tcp100 [11987.363916] LNetError: 474514:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11987.367954] LNetError: 474514:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11987.374640] LNet: Removed LNI 192.168.203.130@tcp101 [11988.383181] LNet: Removed LNI 192.168.203.130@tcp102 [11988.384910] LNetError: 474514:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11988.391111] LNetError: 474514:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11988.394414] LNet: Removed LNI 192.168.203.130@tcp103 [11989.407177] LNet: Removed LNI 192.168.203.130@tcp104 [11990.431156] LNet: Removed LNI 192.168.203.130@tcp105 [11990.433191] LNetError: 474514:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11990.446681] LNetError: 474514:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [11990.451786] LNet: Removed LNI 192.168.203.130@tcp106 [11991.455102] LNet: Removed LNI 192.168.203.130@tcp107 [11992.479148] LNet: Removed LNI 192.168.203.130@tcp108 [11993.503175] LNet: Removed LNI 192.168.203.130@tcp109 [11994.527125] LNet: Removed LNI 192.168.203.130@tcp110 [11994.529298] LNetError: 474514:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11994.536449] LNetError: 474514:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [11994.550464] LNet: Removed LNI 192.168.203.130@tcp111 [11995.615159] LNet: Removed LNI 192.168.203.130@tcp112 [11996.639176] LNet: Removed LNI 192.168.203.130@tcp113 [11997.663465] LNet: Removed LNI 192.168.203.130@tcp114 [11998.687131] LNet: Removed LNI 192.168.203.130@tcp115 [11999.711156] LNet: Removed LNI 192.168.203.130@tcp116 [12000.735690] LNet: Removed LNI 192.168.203.130@tcp117 [12000.740592] LNet: Removed LNI 192.168.203.130@tcp118 [12000.749366] LNet: Removed LNI 192.168.203.130@tcp119 [12000.754809] LNet: Removed LNI 192.168.203.130@tcp120 [12000.757467] LNet: Removed LNI 192.168.203.130@tcp121 [12000.762996] LNet: Removed LNI 192.168.203.130@tcp122 [12000.764988] LNet: Removed LNI 192.168.203.130@tcp123 [12000.767328] LNet: Removed LNI 192.168.203.130@tcp124 [12000.769440] LNet: Removed LNI 192.168.203.130@tcp125 [12000.772347] LNet: Removed LNI 192.168.203.130@tcp126 [12000.775519] LNet: Removed LNI 192.168.203.130@tcp127 [12000.777394] LNet: Removed LNI 192.168.203.130@tcp128 [12000.779223] LNet: Removed LNI 192.168.203.130@tcp129 [12000.781087] LNet: Removed LNI 192.168.203.130@tcp130 [12000.784599] LNet: Removed LNI 192.168.203.130@tcp131 [12000.787590] LNet: Removed LNI 192.168.203.130@tcp132 [12000.789352] LNet: Removed LNI 192.168.203.130@tcp133 [12000.790969] LNet: Removed LNI 192.168.203.130@tcp134 [12000.802780] LNet: Removed LNI 192.168.203.130@tcp135 [12001.469093] Key type .llcrypt unregistered [12001.471515] Key type ._llcrypt unregistered [12018.580721] Key type ._llcrypt registered [12018.582554] Key type .llcrypt registered [12018.658670] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [12030.156703] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [12030.834790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12030.917611] alg: No test for adler32 (adler32-zlib) [12032.001229] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [12032.200349] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [12033.871185] Key type lgssc registered [12035.010640] Lustre: Echo OBD driver; http://www.lustre.org/ [12044.156173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12050.824195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12057.538558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12064.556956] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12071.778042] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 16:07:40 (1763413660) [12087.302977] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [12089.551534] Key type lgssc unregistered [12089.781212] LNet: 478536:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12089.785963] LNetError: 478536:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12089.800990] LNet: Removed LNI 192.168.203.130@tcp [12090.533789] Key type .llcrypt unregistered [12090.535473] Key type ._llcrypt unregistered [12104.598909] Key type ._llcrypt registered [12104.600590] Key type .llcrypt registered [12104.713678] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [12105.413686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12105.422076] alg: No test for adler32 (adler32-zlib) [12106.473621] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [12106.707678] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [12108.383177] Key type lgssc registered [12109.453716] Lustre: Echo OBD driver; http://www.lustre.org/ [12118.371226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12118.447154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12118.467179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12119.725310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12119.779166] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12119.852554] Lustre: lustre-MDT0000: new disk, initializing [12119.929266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12119.945633] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12123.069930] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12132.521443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12132.595222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12132.688589] Lustre: 479720:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12132.745983] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12132.749394] Lustre: Skipped 1 previous similar message [12132.825584] Lustre: lustre-MDT0001: new disk, initializing [12132.946226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12133.019103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12133.035877] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12136.351363] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12142.441811] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12145.400693] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12151.647972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12151.701815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12151.890697] Lustre: lustre-OST0000: new disk, initializing [12151.896649] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12151.976838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12157.190566] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12160.603134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12160.611107] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12160.731664] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12163.795864] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12166.862643] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing lnet_if_list [12171.133595] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [12171.157663] LNet: Added LNI 192.168.203.130@tcp10 [8/256/0/180] [12175.753575] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [12175.779830] LNet: Added LNI 192.168.203.130@tcp11 [8/256/0/180] [12179.300984] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [12179.341070] LNet: Added LNI 192.168.203.130@tcp12 [8/256/0/180] [12182.738627] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [12182.773179] LNet: Added LNI 192.168.203.130@tcp13 [8/256/0/180] [12186.869327] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [12186.897515] LNet: Added LNI 192.168.203.130@tcp14 [8/256/0/180] [12190.586480] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [12190.620299] LNet: Added LNI 192.168.203.130@tcp15 [8/256/0/180] [12194.670611] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [12194.697603] LNet: Added LNI 192.168.203.130@tcp16 [8/256/0/180] [12199.345710] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [12199.387323] LNet: Added LNI 192.168.203.130@tcp17 [8/256/0/180] [12203.431176] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [12203.477157] LNet: Added LNI 192.168.203.130@tcp18 [8/256/0/180] [12207.863869] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [12207.920187] LNet: Added LNI 192.168.203.130@tcp19 [8/256/0/180] [12212.925318] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [12212.955647] LNet: Added LNI 192.168.203.130@tcp20 [8/256/0/180] [12218.414995] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [12218.453305] LNet: Added LNI 192.168.203.130@tcp21 [8/256/0/180] [12226.361725] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [12226.431209] LNet: Added LNI 192.168.203.130@tcp22 [8/256/0/180] [12233.100453] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [12233.142540] LNet: Added LNI 192.168.203.130@tcp23 [8/256/0/180] [12239.991813] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [12240.029161] LNet: Added LNI 192.168.203.130@tcp24 [8/256/0/180] [12245.686350] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [12245.732217] LNet: Added LNI 192.168.203.130@tcp25 [8/256/0/180] [12251.937157] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [12251.984508] LNet: Added LNI 192.168.203.130@tcp26 [8/256/0/180] [12258.978908] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [12259.042270] LNet: Added LNI 192.168.203.130@tcp27 [8/256/0/180] [12259.048854] LNetError: 478983:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [12259.060803] LNetError: Protocol error connecting to 192.168.203.130@tcp26 at host 192.168.203.130:988: is it running a compatible version of Lustre? [12265.151856] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [12265.184631] LNet: Added LNI 192.168.203.130@tcp28 [8/256/0/180] [12265.192177] LNetError: 478982:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [12265.206595] LNetError: Protocol error connecting to 192.168.203.130@tcp26 at host 192.168.203.130:988: is it running a compatible version of Lustre? [12270.531942] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [12270.589190] LNet: Added LNI 192.168.203.130@tcp29 [8/256/0/180] [12270.600364] LNetError: 478980:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [12270.612893] LNetError: Protocol error connecting to 192.168.203.130@tcp28 at host 192.168.203.130:988: is it running a compatible version of Lustre? [12270.818584] LNet: 1 local NIs in recovery (showing 1): 192.168.203.130@tcp28 [12272.865342] LNetError: 478982:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [12272.874182] LNetError: 478982:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 2 previous similar messages [12272.886922] LNetError: Protocol error connecting to 192.168.203.130@tcp29 at host 192.168.203.130:988: is it running a compatible version of Lustre? [12272.912675] LNetError: Skipped 2 previous similar messages [12275.922141] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [12275.938884] LNet: Added LNI 192.168.203.130@tcp30 [8/256/0/180] [12278.000675] LNetError: 478981:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [12278.014690] LNetError: 478981:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 2 previous similar messages [12278.033409] LNetError: Protocol error connecting to 192.168.203.130@tcp28 at host 192.168.203.130:988: is it running a compatible version of Lustre? [12278.074315] LNetError: Skipped 2 previous similar messages [12282.248299] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [12282.296907] LNet: Added LNI 192.168.203.130@tcp31 [8/256/0/180] [12286.180398] LNetError: 478980:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [12286.202350] LNetError: 478980:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 5 previous similar messages [12286.221435] LNetError: Protocol error connecting to 192.168.203.130@tcp31 at host 192.168.203.130:988: is it running a compatible version of Lustre? [12286.240094] LNetError: Skipped 5 previous similar messages [12287.841939] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [12287.868258] LNet: Added LNI 192.168.203.130@tcp32 [8/256/0/180] [12293.621063] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [12293.643646] LNet: Added LNI 192.168.203.130@tcp33 [8/256/0/180] [12299.907267] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [12299.928936] LNet: Added LNI 192.168.203.130@tcp34 [8/256/0/180] [12303.585599] LNetError: 478983:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [12303.596826] LNetError: 478983:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 12 previous similar messages [12303.604583] LNetError: Protocol error connecting to 192.168.203.130@tcp29 at host 192.168.203.130:988: is it running a compatible version of Lustre? [12303.616110] LNetError: Skipped 12 previous similar messages [12305.214546] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [12305.235590] LNet: Added LNI 192.168.203.130@tcp35 [8/256/0/180] [12310.857104] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [12310.923287] LNet: Added LNI 192.168.203.130@tcp36 [8/256/0/180] [12316.582138] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [12316.596219] LNet: Added LNI 192.168.203.130@tcp37 [8/256/0/180] [12322.864647] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [12322.918543] LNet: Added LNI 192.168.203.130@tcp38 [8/256/0/180] [12329.056582] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [12329.094680] LNet: Added LNI 192.168.203.130@tcp39 [8/256/0/180] [12331.260329] LNet: 10 local NIs in recovery (showing 10): 192.168.203.130@tcp37, 192.168.203.130@tcp38, 192.168.203.130@tcp36, 192.168.203.130@tcp34, 192.168.203.130@tcp35, 192.168.203.130@tcp33, 192.168.203.130@tcp31, 192.168.203.130@tcp30, 192.168.203.130@tcp29, 192.168.203.130@tcp28 [12335.205100] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [12335.230485] LNet: Added LNI 192.168.203.130@tcp40 [8/256/0/180] [12336.416675] LNetError: 478983:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [12336.434975] LNetError: 478983:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 35 previous similar messages [12336.446448] LNetError: Protocol error connecting to 192.168.203.130@tcp33 at host 192.168.203.130:988: is it running a compatible version of Lustre? [12336.457821] LNetError: Skipped 35 previous similar messages [12341.216516] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [12341.246034] LNet: Added LNI 192.168.203.130@tcp41 [8/256/0/180] [12346.850317] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [12346.873188] LNet: Added LNI 192.168.203.130@tcp42 [8/256/0/180] [12352.786878] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [12352.810815] LNet: Added LNI 192.168.203.130@tcp43 [8/256/0/180] [12359.109371] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [12359.127524] LNet: Added LNI 192.168.203.130@tcp44 [8/256/0/180] [12365.550958] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [12365.577545] LNet: Added LNI 192.168.203.130@tcp45 [8/256/0/180] [12372.424189] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [12372.442031] LNet: Added LNI 192.168.203.130@tcp46 [8/256/0/180] [12379.257719] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [12379.282581] LNet: Added LNI 192.168.203.130@tcp47 [8/256/0/180] [12386.670503] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [12386.707027] LNet: Added LNI 192.168.203.130@tcp48 [8/256/0/180] [12393.861784] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [12393.895214] LNet: Added LNI 192.168.203.130@tcp49 [8/256/0/180] [12401.154390] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [12401.176450] LNet: Added LNI 192.168.203.130@tcp50 [8/256/0/180] [12401.189796] LNetError: 478980:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.130 [12401.193569] LNetError: 478980:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 69 previous similar messages [12401.197052] LNetError: Protocol error connecting to 192.168.203.130@tcp49 at host 192.168.203.130:988: is it running a compatible version of Lustre? [12401.202233] LNetError: Skipped 69 previous similar messages [12422.116512] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12422.126722] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12422.141104] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12422.166512] Lustre: Skipped 1 previous similar message [12424.710111] LustreError: 504005:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [12424.873723] Lustre: server umount lustre-OST0000 complete [12429.792828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12429.799882] LustreError: Skipped 1 previous similar message [12429.803374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12429.813121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12429.815859] Lustre: Skipped 1 previous similar message [12432.362524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12432.363153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12434.230402] LustreError: 504207:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [12434.233304] LustreError: 504207:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12434.385950] Lustre: server umount lustre-MDT0000 complete [12437.842284] LustreError: 479714:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763414027 with bad export cookie 5117711294253274673 [12437.850730] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12437.860850] LustreError: 479714:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12438.015650] LustreError: 504409:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12438.019675] LustreError: 504409:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12438.171585] Lustre: server umount lustre-MDT0001 complete [12446.394699] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [12448.825104] Key type lgssc unregistered [12449.178066] LNet: 504948:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12449.186382] LNetError: 504948:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12449.195113] LNet: Removed LNI 192.168.203.130@tcp [12450.207636] LNet: Removed LNI 192.168.203.130@tcp10 [12450.213156] LNetError: 504948:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12450.218356] LNetError: 504948:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12451.231223] LNet: Removed LNI 192.168.203.130@tcp11 [12451.234703] LNetError: 504948:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12451.249865] LNet: Removed LNI 192.168.203.130@tcp12 [12452.319758] LNet: Removed LNI 192.168.203.130@tcp13 [12453.343331] LNet: Removed LNI 192.168.203.130@tcp14 [12453.345114] LNetError: 504948:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12453.359079] LNetError: 504948:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [12453.364795] LNet: Removed LNI 192.168.203.130@tcp15 [12454.431298] LNet: Removed LNI 192.168.203.130@tcp16 [12455.455185] LNet: Removed LNI 192.168.203.130@tcp17 [12456.480094] LNet: Removed LNI 192.168.203.130@tcp18 [12457.503159] LNet: Removed LNI 192.168.203.130@tcp19 [12457.505244] LNetError: 504948:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12457.515841] LNetError: 504948:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [12457.527887] LNet: Removed LNI 192.168.203.130@tcp20 [12458.591227] LNet: Removed LNI 192.168.203.130@tcp21 [12459.615561] LNet: Removed LNI 192.168.203.130@tcp22 [12460.639173] LNet: Removed LNI 192.168.203.130@tcp23 [12461.663225] LNet: Removed LNI 192.168.203.130@tcp24 [12462.687162] LNet: Removed LNI 192.168.203.130@tcp25 [12463.711402] LNet: Removed LNI 192.168.203.130@tcp26 [12464.735275] LNet: Removed LNI 192.168.203.130@tcp27 [12464.739088] LNet: Removed LNI 192.168.203.130@tcp28 [12464.752449] LNet: Removed LNI 192.168.203.130@tcp29 [12465.759268] LNet: Removed LNI 192.168.203.130@tcp30 [12465.765635] LNetError: 504948:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12465.769815] LNetError: 504948:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 10 previous similar messages [12465.773366] LNet: Removed LNI 192.168.203.130@tcp31 [12466.783253] LNet: Removed LNI 192.168.203.130@tcp32 [12466.785383] LNet: Removed LNI 192.168.203.130@tcp33 [12466.796683] LNet: Removed LNI 192.168.203.130@tcp34 [12467.808557] LNet: Removed LNI 192.168.203.130@tcp35 [12468.831178] LNet: Removed LNI 192.168.203.130@tcp36 [12468.832912] LNet: Removed LNI 192.168.203.130@tcp37 [12468.841674] LNet: Removed LNI 192.168.203.130@tcp38 [12469.856267] LNet: Removed LNI 192.168.203.130@tcp39 [12470.879202] LNet: Removed LNI 192.168.203.130@tcp40 [12470.887863] LNet: Removed LNI 192.168.203.130@tcp41 [12470.893881] LNet: Removed LNI 192.168.203.130@tcp42 [12470.898410] LNet: Removed LNI 192.168.203.130@tcp43 [12470.909772] LNet: Removed LNI 192.168.203.130@tcp44 [12470.915801] LNet: Removed LNI 192.168.203.130@tcp45 [12471.967187] LNet: Removed LNI 192.168.203.130@tcp46 [12471.970738] LNet: Removed LNI 192.168.203.130@tcp47 [12472.991404] LNet: Removed LNI 192.168.203.130@tcp48 [12472.993623] LNet: Removed LNI 192.168.203.130@tcp49 [12473.006216] LNet: Removed LNI 192.168.203.130@tcp50 [12473.612947] LNetError: 505011:0:(module.c:692:libcfs_exit()) Portals memory leaked: 752 bytes [12473.650559] Key type .llcrypt unregistered [12473.656562] Key type ._llcrypt unregistered [12490.834624] Key type ._llcrypt registered [12490.836348] Key type .llcrypt registered [12490.976938] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [12491.960278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12491.999675] alg: No test for adler32 (adler32-zlib) [12493.138460] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [12493.446821] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [12495.119288] Key type lgssc registered [12496.976225] Lustre: Echo OBD driver; http://www.lustre.org/ [12505.333531] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12505.358932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12506.909279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12510.699079] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12516.321343] LustreError: 506202:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12516.344694] LustreError: 506202:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12519.173822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12519.669683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12524.044885] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12529.720829] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12533.487114] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12539.876247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12540.084171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12544.790858] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12550.000086] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12562.641655] loop0: detected capacity change from 0 to 204800000 [12562.800088] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12562.808119] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12562.818277] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12562.891181] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12568.292881] loop0: detected capacity change from 0 to 204800000 [12568.549331] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12568.574581] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12568.586057] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12568.740549] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12574.349231] loop0: detected capacity change from 0 to 204800000 [12579.228641] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12579.288971] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12579.435113] Lustre: fs15246-OST0000: new disk, initializing [12579.451588] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [12579.636268] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12585.692966] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12594.705622] loop1: detected capacity change from 0 to 204800000 [12600.883036] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12600.997965] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12601.102434] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [12601.164922] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [12601.284578] Lustre: fs15246-MDT0000: new disk, initializing [12601.476470] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12601.499325] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12601.642214] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12602.720911] Lustre: 510581:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000fe6597db x1849073685540224/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763414208 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [12605.917832] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12611.874079] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12611.884599] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12611.913330] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12630.520390] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12630.540805] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [12632.572703] LustreError: 511704:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [12632.638684] Lustre: server umount fs15246-MDT0000 complete [12635.618163] LustreError: 507622:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12639.835173] LustreError: 512206:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [12639.847726] LustreError: 512206:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12639.870039] Lustre: server umount fs15246-OST0000 complete [12647.906069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12647.911686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12647.925597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12648.930136] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12648.940563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12648.975367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12650.984484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12650.987259] Lustre: Skipped 1 previous similar message [12656.097342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12656.118458] Lustre: Skipped 1 previous similar message [12662.239152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12662.280117] LustreError: 512709:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12662.287122] LustreError: 512709:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [12662.378704] Lustre: server umount lustre-OST0000 complete [12666.339512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12666.357240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12666.359289] Lustre: Skipped 3 previous similar messages [12671.565890] LustreError: 512912:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12671.587532] LustreError: 512912:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12671.720385] Lustre: server umount lustre-MDT0000 complete [12674.975457] LustreError: 506185:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763414264 with bad export cookie 11611744738587356265 [12674.981124] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12674.989945] LustreError: 506185:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [12675.257175] Lustre: server umount lustre-MDT0001 complete [12683.872757] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [12686.559805] Key type lgssc unregistered [12686.818721] LNet: 513644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12686.833192] LNetError: 513644:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12686.851740] LNet: Removed LNI 192.168.203.130@tcp [12687.802874] Key type .llcrypt unregistered [12687.804295] Key type ._llcrypt unregistered [12703.423672] Key type ._llcrypt registered [12703.425832] Key type .llcrypt registered [12703.490504] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [12704.182989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12704.234842] alg: No test for adler32 (adler32-zlib) [12705.205244] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [12705.408944] LNet: Added LNI 192.168.203.130@tcp [8/256/0/180] [12707.071189] Key type lgssc registered [12708.069218] Lustre: Echo OBD driver; http://www.lustre.org/ [12716.007336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12721.324083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12727.440367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12732.892080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12749.985193] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [12757.581986] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [12766.185330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12771.545811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12776.634049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12781.895521] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12789.655546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12789.698255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12789.713906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12790.963273] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12791.006706] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12791.111182] Lustre: lustre-MDT0000: new disk, initializing [12791.280324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12791.321599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12794.991803] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12806.902605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12807.001101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12807.186532] Lustre: 518073:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12807.282136] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12807.284966] Lustre: Skipped 1 previous similar message [12807.373385] Lustre: lustre-MDT0001: new disk, initializing [12807.497936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12807.609572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12807.629852] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12811.445658] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12817.805506] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12821.024240] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12826.740530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12826.828224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12826.996536] Lustre: lustre-OST0000: new disk, initializing [12827.005261] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12827.078024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12832.165646] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_default_debug -1 all [12835.406679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12835.418376] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12835.546517] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12838.407205] Lustre: DEBUG MARKER: oleg330-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12842.111866] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12842.270230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12845.838888] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12846.141907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12851.171596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12851.173749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12851.176222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12851.184984] Lustre: Skipped 1 previous similar message [12853.644655] LustreError: 521018:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [12853.857280] Lustre: server umount lustre-OST0000 complete [12858.854913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12858.858486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12858.876410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12858.879527] Lustre: Skipped 1 previous similar message [12861.413726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12861.416839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12863.885294] LustreError: 521219:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [12863.891225] LustreError: 521219:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12864.114245] Lustre: server umount lustre-MDT0000 complete [12867.740102] LustreError: 518065:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763414457 with bad export cookie 16936600702339725405 [12867.749823] LustreError: MGC192.168.203.130@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12867.759314] LustreError: 518065:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [12867.939917] LustreError: 521421:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12867.951829] LustreError: 521421:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12868.232555] Lustre: server umount lustre-MDT0001 complete [12888.101520] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing set_hostid [12896.144431] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing load_modules_local [12905.885855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12912.416379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12919.091447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12924.079257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12925.457292] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12633 sec ========= 16:21:54 (1763414514) [12926.864745] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:21:56 (1763414516) === [12929.668183] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:21:58 (1763414518) === [12963.106543] Lustre: DEBUG MARKER: oleg330-server.virtnet: executing unload_modules_local [12965.652755] Key type lgssc unregistered [12965.945400] LNet: 526277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12965.972637] LNetError: 526277:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12966.002804] LNet: Removed LNI 192.168.203.130@tcp [12966.760165] Key type .llcrypt unregistered [12966.762065] Key type ._llcrypt unregistered