[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 479818683 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002451] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006894] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008007] pid_max: default: 32768 minimum: 301 [ 0.009159] LSM: Security Framework initializing [ 0.010062] Yama: becoming mindful. [ 0.011029] SELinux: Initializing. [ 0.012059] *** VALIDATE selinux *** [ 0.020519] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025121] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026158] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027085] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028120] *** VALIDATE tmpfs *** [ 0.030078] *** VALIDATE proc *** [ 0.031000] *** VALIDATE cgroup *** [ 0.031000] *** VALIDATE cgroup2 *** [ 0.031270] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032299] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034035] Spectre V2 : User space: Vulnerable [ 0.035000] Speculative Store Bypass: Vulnerable [ 0.037534] debug: unmapping init [mem 0xffffffff86859000-0xffffffff86860fff] [ 0.039973] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040659] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041022] ... version: 2 [ 0.042011] ... bit width: 48 [ 0.043010] ... generic registers: 4 [ 0.044007] ... value mask: 0000ffffffffffff [ 0.045008] ... max period: 00007fffffffffff [ 0.046008] ... fixed-purpose events: 3 [ 0.047008] ... event mask: 000000070000000f [ 0.049229] rcu: Hierarchical SRCU implementation. [ 0.051505] smp: Bringing up secondary CPUs ... [ 0.052507] x86: Booting SMP configuration: [ 0.053019] .... node #0, CPUs: #1 #2 #3 [ 0.057035] smp: Brought up 1 node, 4 CPUs [ 0.059012] smpboot: Max logical packages: 1 [ 0.060008] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143142] node 0 deferred pages initialised in 82ms [ 0.148323] devtmpfs: initialized [ 0.149329] x86/mm: Memory block size: 128MB [ 0.154420] gcov: version magic: 0x41383552 [ 0.159101] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.160082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.164018] pinctrl core: initialized pinctrl subsystem [ 0.166232] [ 0.167014] ************************************************************* [ 0.169012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172009] ** ** [ 0.176011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.178014] ** ** [ 0.183027] ** This means that this kernel is built to expose internal ** [ 0.186014] ** IOMMU data structures, which may compromise security on ** [ 0.189013] ** your system. ** [ 0.191012] ** ** [ 0.194023] ** If you see this message and you are not debugging the ** [ 0.196008] ** kernel, report this immediately to your vendor! ** [ 0.198016] ** ** [ 0.200010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.202018] ************************************************************* [ 0.206789] NET: Registered protocol family 16 [ 0.209927] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.213074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.214079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.217460] cpuidle: using governor menu [ 0.218706] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.221575] PCI: Using configuration type 1 for base access [ 0.224146] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.236081] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.239022] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.246381] cryptd: max_cpu_qlen set to 1000 [ 0.248390] ACPI: Added _OSI(Module Device) [ 0.250022] ACPI: Added _OSI(Processor Device) [ 0.251015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.253014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.256961] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.265574] ACPI: Interpreter enabled [ 0.266078] ACPI: PM: (supports S0 S3 S4 S5) [ 0.268015] ACPI: Using IOAPIC for interrupt routing [ 0.269167] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.272446] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.281482] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.284056] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.286021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.289102] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.293407] acpiphp: Slot [2] registered [ 0.294107] acpiphp: Slot [5] registered [ 0.296106] acpiphp: Slot [6] registered [ 0.298172] acpiphp: Slot [7] registered [ 0.299159] acpiphp: Slot [8] registered [ 0.300254] acpiphp: Slot [9] registered [ 0.302160] acpiphp: Slot [10] registered [ 0.303162] acpiphp: Slot [3] registered [ 0.305093] acpiphp: Slot [4] registered [ 0.306104] acpiphp: Slot [11] registered [ 0.308540] acpiphp: Slot [12] registered [ 0.309138] acpiphp: Slot [13] registered [ 0.311140] acpiphp: Slot [14] registered [ 0.313168] acpiphp: Slot [15] registered [ 0.314120] acpiphp: Slot [16] registered [ 0.316670] acpiphp: Slot [17] registered [ 0.318132] acpiphp: Slot [18] registered [ 0.319099] acpiphp: Slot [19] registered [ 0.321134] acpiphp: Slot [20] registered [ 0.322148] acpiphp: Slot [21] registered [ 0.324123] acpiphp: Slot [22] registered [ 0.326113] acpiphp: Slot [23] registered [ 0.328168] acpiphp: Slot [24] registered [ 0.329160] acpiphp: Slot [25] registered [ 0.330139] acpiphp: Slot [26] registered [ 0.333063] acpiphp: Slot [27] registered [ 0.334116] acpiphp: Slot [28] registered [ 0.337291] acpiphp: Slot [29] registered [ 0.338361] acpiphp: Slot [30] registered [ 0.341132] acpiphp: Slot [31] registered [ 0.342083] PCI host bridge to bus 0000:00 [ 0.342980] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.345028] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.347028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.349033] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.351036] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.352028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.354195] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.357493] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.360574] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.369017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.373062] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.375027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.378034] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.380031] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.383118] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.385862] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.388053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.391893] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.397023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.408017] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.413024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.419753] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.423015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.431033] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.447033] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.459673] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.469028] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.475140] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.493044] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.505000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.510018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.515019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.532042] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.547228] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.554023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.562019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.579019] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.588000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.594016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.600015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.614015] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.627163] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.635013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.640023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.657031] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.672000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.675550] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.679528] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.682686] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.684204] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.690071] iommu: Default domain type: Passthrough [ 0.691511] SCSI subsystem initialized [ 0.692182] ACPI: bus type USB registered [ 0.694112] usbcore: registered new interface driver usbfs [ 0.697119] usbcore: registered new interface driver hub [ 0.700233] usbcore: registered new device driver usb [ 0.703227] pps_core: LinuxPPS API ver. 1 registered [ 0.706018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.710075] PTP clock support registered [ 0.713229] EDAC MC: Ver: 3.0.0 [ 0.716300] PCI: Using ACPI for IRQ routing [ 0.717504] NetLabel: Initializing [ 0.718013] NetLabel: domain hash size = 128 [ 0.721014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.722000] NetLabel: unlabeled traffic allowed by default [ 0.726158] vgaarb: loaded [ 0.728488] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.731026] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.739000] clocksource: Switched to clocksource kvm-clock [ 0.885876] VFS: Disk quotas dquot_6.6.0 [ 0.887442] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.889468] *** VALIDATE ramfs *** [ 0.890285] *** VALIDATE hugetlbfs *** [ 0.891250] pnp: PnP ACPI init [ 0.893992] pnp: PnP ACPI: found 6 devices [ 0.914129] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.917897] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.920058] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.922361] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.925189] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.927908] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 0.930930] NET: Registered protocol family 2 [ 0.933382] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.937387] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.940583] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.947555] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.950711] TCP: Hash tables configured (established 65536 bind 65536) [ 0.953561] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.956358] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.959543] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.962680] NET: Registered protocol family 1 [ 0.967703] RPC: Registered named UNIX socket transport module. [ 0.973706] RPC: Registered udp transport module. [ 0.977094] RPC: Registered tcp transport module. [ 0.979317] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.982138] NET: Registered protocol family 44 [ 0.983885] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.985877] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.987797] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.992928] PCI: CLS 0 bytes, default 64 [ 0.997071] Unpacking initramfs... [ 2.760150] debug: unmapping init [mem 0xffff9086fcc54000-0xffff9086fffbffff] [ 2.763802] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.765905] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.769178] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.338942] Initialise system trusted keyrings [ 3.340584] Key type blacklist registered [ 3.343699] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.352519] zbud: loaded [ 3.357092] *** VALIDATE nfs *** [ 3.358275] *** VALIDATE nfs4 *** [ 3.359830] pstore: using deflate compression [ 3.363479] Platform Keyring initialized [ 3.527791] NET: Registered protocol family 38 [ 3.533413] Key type asymmetric registered [ 3.534790] Asymmetric key parser 'x509' registered [ 3.537268] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.541143] io scheduler mq-deadline registered [ 3.542836] io scheduler kyber registered [ 3.544659] io scheduler bfq registered [ 3.546461] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.549755] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.553865] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.561447] ACPI: Power Button [PWRF] [ 3.690411] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.783708] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.985620] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.098986] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.333940] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.363648] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.401523] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.406107] Non-volatile memory driver v1.3 [ 4.407636] Linux agpgart interface v0.103 [ 4.461967] virtio_blk virtio1: [vda] 133928 512-byte logical blocks (68.6 MB/65.4 MiB) [ 4.472439] vda: detected capacity change from 0 to 68571136 [ 4.493079] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.496238] vdb: detected capacity change from 0 to 1073741824 [ 4.515805] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.518761] vdc: detected capacity change from 0 to 2621440000 [ 4.550407] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.555519] vdd: detected capacity change from 0 to 2621440000 [ 4.586993] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.591540] vde: detected capacity change from 0 to 4294967296 [ 4.609154] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.614688] vdf: detected capacity change from 0 to 4294967296 [ 4.626424] libphy: Fixed MDIO Bus: probed [ 4.635761] usbcore: registered new interface driver usbserial_generic [ 4.640937] usbserial: USB Serial support registered for generic [ 4.645418] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.650226] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.652601] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.655455] mousedev: PS/2 mouse device common for all mice [ 4.662300] rtc_cmos 00:05: RTC can wake from S4 [ 4.666479] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.672741] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.678910] rtc_cmos 00:05: registered as rtc0 [ 4.689254] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.695397] intel_pstate: CPU model not supported [ 4.699741] hid: raw HID events driver (C) Jiri Kosina [ 4.702157] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.702295] usbcore: registered new interface driver usbhid [ 4.707347] usbhid: USB HID core driver [ 4.709789] drop_monitor: Initializing network drop monitor service [ 4.712896] Initializing XFRM netlink socket [ 4.715179] NET: Registered protocol family 10 [ 4.719249] Segment Routing with IPv6 [ 4.720604] NET: Registered protocol family 17 [ 4.724132] mpls_gso: MPLS GSO support [ 4.728924] RAS: Correctable Errors collector initialized. [ 4.730542] AVX version of gcm_enc/dec engaged. [ 4.732378] AES CTR mode by8 optimization enabled [ 4.856990] sched_clock: Marking stable (4856783062, 0)->(5751549418, -894766356) [ 4.860519] registered taskstats version 1 [ 4.863924] Loading compiled-in X.509 certificates [ 4.873753] zswap: loaded using pool lzo/zbud [ 4.916975] Key type big_key registered [ 4.942081] Key type encrypted registered [ 4.943839] ima: No TPM chip found, activating TPM-bypass! [ 4.946238] ima: Allocated hash algorithm: sha1 [ 4.947782] ima: No architecture policies found [ 4.949330] evm: Initialising EVM extended attributes: [ 4.951420] evm: security.selinux [ 4.952741] evm: security.ima [ 4.953943] evm: security.capability [ 4.955423] evm: HMAC attrs: 0x1 [ 4.958302] rtc_cmos 00:05: setting system clock to 2025-12-03 06:49:46 UTC (1764744586) [ 4.966364] debug: unmapping init [mem 0xffffffff87803000-0xffffffff879fffff] [ 4.969614] debug: unmapping init [mem 0xffffffff86582000-0xffffffff86858fff] [ 4.976188] Write protecting the kernel read-only data: 28672k [ 4.980272] debug: unmapping init [mem 0xffffffff84c03000-0xffffffff84dfffff] [ 4.983549] debug: unmapping init [mem 0xffffffff85514000-0xffffffff855fffff] [ 5.030349] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.044426] systemd[1]: Detected virtualization kvm. [ 5.049782] systemd[1]: Detected architecture x86-64. [ 5.054410] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.088267] systemd[1]: No hostname configured. [ 5.090694] systemd[1]: Set hostname to . [ 5.092791] random: systemd: uninitialized urandom read (16 bytes read) [ 5.096698] systemd[1]: Initializing machine ID from random generator. [ 5.320781] random: systemd: uninitialized urandom read (16 bytes read) [ 5.323392] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 5.328298] random: systemd: uninitialized urandom read (16 bytes read) [ 5.331852] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 5.338828] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Reached target Slices. Starting Journal Service... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.562477] device-mapper: uevent: version 1.0.3 [ 6.565288] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ 7.727731] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target Basic System.[ 7.752062] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.173844] scsi host0: ata_piix [ 8.975753] scsi host1: ata_piix [ 8.978320] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.981470] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.468512] random: crng init done [ 13.470225] random: 7 urandom warning(s) missed due to ratelimiting [ 14.822095] hrtimer: interrupt took 64927384 ns [ 14.990952] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 16.517881] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.867714] printk: systemd: 26 output lines suppressed due to ratelimiting [ 19.517161] SELinux: Disabled at runtime. [ 19.619762] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.629894] systemd[1]: Detected virtualization kvm. [ 19.631966] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 20.883399] systemd[1]: initrd-switch-root.service: Succeeded. [ 20.892117] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.907381] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.913486] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.918770] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.936441] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.953934] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. Mounting Huge Pages File System... Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ 21.365117] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 21.833622] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.487146] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.570543] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.847668] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.869728] EDAC sbridge: Ver: 1.1.2 [ 26.787284] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (6s / no limit)[ 27.652256] NFS: Registering the id_resolver key type [ 27.657807] Key type id_resolver registered [ 27.659268] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. Starting Hostname Service... [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg137-server login: [ 55.852827] libcfs: loading out-of-tree module taints kernel. [ 55.869728] Key type ._llcrypt registered [ 55.871368] Key type .llcrypt registered [ 55.914396] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 63.444218] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 64.168887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 64.176861] alg: No test for adler32 (adler32-zlib) [ 65.160765] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 65.471385] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 67.087223] Key type lgssc registered [ 67.850150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 75.236772] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 75.911295] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 79.265402] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 81.944071] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 84.652261] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 90.097503] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 94.757212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 94.778567] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 94.786920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 95.879304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 95.892144] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 95.931818] Lustre: lustre-MDT0000: new disk, initializing [ 95.958493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 95.965219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 97.305354] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 102.636216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 102.660673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 102.688306] Lustre: 6486:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 102.700453] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 102.702823] Lustre: Skipped 1 previous similar message [ 102.742728] Lustre: lustre-MDT0001: new disk, initializing [ 102.764316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 102.773038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 102.776420] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 104.179743] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 106.590841] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 110.100029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 110.124522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 110.219770] Lustre: lustre-OST0000: new disk, initializing [ 110.221647] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 110.237502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 112.221153] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 116.716996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 116.720063] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 116.747833] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 117.772460] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 117.799188] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 117.840254] Lustre: lustre-OST0001: new disk, initializing [ 117.842966] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 117.862940] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 119.947298] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 122.350580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 122.354717] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 122.371653] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 126.226502] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 128.923213] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 136.200512] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing check_logdir /tmp/testlogs/ [ 138.074188] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing yml_node [ 139.995848] Lustre: DEBUG MARKER: Client: 2.17.0.RC1 [ 141.009892] Lustre: DEBUG MARKER: MDS: 2.17.0.RC1 [ 141.990853] Lustre: DEBUG MARKER: OSS: 2.17.0.RC1 [ 142.632536] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Dec 3 01:52:03 EST 2025 [ 150.537578] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 151.214870] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 164.319781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 164.322395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 164.328434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 168.416205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 168.416752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 168.420274] Lustre: Skipped 1 previous similar message [ 168.425627] Lustre: Skipped 1 previous similar message [ 169.544159] LustreError: 12187:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 169.592136] Lustre: server umount lustre-MDT0000 complete [ 171.236092] LustreError: 9451:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764744752 with bad export cookie 2289107127352234310 [ 171.238249] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 171.240635] LustreError: 9451:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 173.537335] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 173.537791] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 173.542497] Lustre: Skipped 2 previous similar messages [ 173.545341] Lustre: Skipped 2 previous similar messages [ 176.543953] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 176.547101] Lustre: Skipped 1 previous similar message [ 177.346377] LustreError: 12389:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 177.349975] LustreError: 12389:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 177.408377] Lustre: server umount lustre-MDT0001 complete [ 179.250800] LustreError: 12590:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 179.253480] LustreError: 12590:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 179.275942] Lustre: server umount lustre-OST0000 complete [ 181.098100] Lustre: server umount lustre-OST0001 complete [ 184.066855] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 188.294125] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 192.941871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 196.036383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 199.115983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 202.195360] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 205.703719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 205.741925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 205.880044] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 205.904176] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 205.949321] Lustre: lustre-MDT0000: new disk, initializing [ 205.996285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 206.006493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 207.549373] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 212.204224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 212.227403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 212.268173] Lustre: 14970:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 212.383438] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 213.794426] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 216.953318] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 218.192443] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 220.697053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 220.722921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 220.865036] Lustre: lustre-OST0000: new disk, initializing [ 220.866849] Lustre: Skipped 1 previous similar message [ 220.868433] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 220.869853] Lustre: Skipped 2 previous similar messages [ 220.899471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 220.902398] Lustre: Skipped 1 previous similar message [ 222.525320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 222.528527] Lustre: Skipped 1 previous similar message [ 222.530649] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 222.575549] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 222.928931] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 226.213606] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 227.618685] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 227.699801] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 229.064405] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 229.150396] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 232.930026] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 232.930307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 232.931354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 232.932805] LustreError: Skipped 1 previous similar message [ 232.938733] Lustre: Skipped 1 previous similar message [ 243.170833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 243.172517] Lustre: Skipped 4 previous similar messages [ 244.191130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 244.209099] LustreError: 17910:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 244.212472] LustreError: 17910:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 244.256919] Lustre: server umount lustre-OST0000 complete [ 248.288812] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 248.289085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 248.294806] Lustre: Skipped 1 previous similar message [ 251.585343] LustreError: 18112:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 251.587670] LustreError: 18112:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 251.620669] Lustre: server umount lustre-MDT0000 complete [ 252.885208] LustreError: 15752:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764744834 with bad export cookie 2289107127352235843 [ 252.889137] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 252.889584] LustreError: 15752:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 253.007323] Lustre: server umount lustre-MDT0001 complete [ 254.549265] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:53:55 (1764744835) [ 255.445438] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 255.996320] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:53:56 (1764744836) [ 258.210800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 258.356114] LustreError: 18982:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 258.363031] LustreError: 18982:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 258.381290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 259.679406] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 262.419981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 263.804377] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 265.516647] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 266.653967] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 268.953570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 270.971801] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 272.710942] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 293.857548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.857936] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 293.858556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 293.858559] Lustre: Skipped 2 previous similar messages [ 293.863565] Lustre: Skipped 1 previous similar message [ 293.865427] LustreError: Skipped 1 previous similar message [ 296.878105] LustreError: 21252:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 296.880922] LustreError: 21252:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 296.923145] Lustre: server umount lustre-OST0000 complete [ 298.464132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 304.415689] Lustre: server umount lustre-MDT0000 complete [ 305.714343] LustreError: 19702:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764744887 with bad export cookie 2289107127352236725 [ 305.718050] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 305.718422] LustreError: 19702:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 305.842655] Lustre: server umount lustre-MDT0001 complete [ 309.339646] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 310.420355] Key type lgssc unregistered [ 310.554474] LNet: 22189:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.556891] LNetError: 22189:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 310.566435] LNet: Removed LNI 192.168.201.137@tcp [ 310.905186] Key type .llcrypt unregistered [ 310.906875] Key type ._llcrypt unregistered [ 319.736274] Key type ._llcrypt registered [ 319.737346] Key type .llcrypt registered [ 319.774369] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 320.294062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 320.305486] alg: No test for adler32 (adler32-zlib) [ 321.170784] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 321.254557] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 322.831144] Key type lgssc registered [ 323.199878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 326.122378] loop: module loaded [ 326.144399] loop0: detected capacity change from 0 to 204800000 [ 326.222246] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 326.228581] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 326.363020] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 326.384296] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 328.326966] loop0: detected capacity change from 0 to 204800000 [ 328.382865] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 328.387315] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 328.391849] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 328.433835] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 330.025748] loop0: detected capacity change from 0 to 204800000 [ 331.960795] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 331.980757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 331.984984] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 333.085212] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 333.104064] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 333.149682] Lustre: lustre-MDT0000: new disk, initializing [ 333.177961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 333.186721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 334.603841] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 338.340231] loop1: detected capacity change from 0 to 204800000 [ 340.272075] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 340.291701] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 340.405401] Lustre: lustre-OST07c6: new disk, initializing [ 340.408502] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 340.411497] Lustre: Skipped 1 previous similar message [ 340.437620] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 342.487908] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 346.628127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 346.632314] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 346.643865] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 350.134453] Lustre: 24323:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 350.139545] Lustre: 24323:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 350.142705] Lustre: 24323:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 350.145971] Lustre: 24323:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 350.149936] Lustre: 24323:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 350.152143] Lustre: 24323:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 361.514113] Lustre: 24323:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 361.518065] Lustre: 24323:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 361.521064] Lustre: 24323:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 361.524582] Lustre: 24323:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 361.528197] Lustre: 24323:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 361.531738] Lustre: 24323:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 365.089892] Lustre: 24323:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 365.093587] Lustre: 24323:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 365.096186] Lustre: 24323:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 365.098371] Lustre: 24323:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 365.100817] Lustre: 24323:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 365.103292] Lustre: 24323:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 365.105673] Lustre: 24323:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 365.108717] Lustre: 24323:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 365.112203] Lustre: 24323:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 365.115077] Lustre: 24323:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 365.117979] Lustre: 24323:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 365.120438] Lustre: 24323:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 368.187939] Lustre: 24323:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 368.192109] Lustre: 24323:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 368.195334] Lustre: 24323:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 368.199046] Lustre: 24323:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 368.202675] Lustre: 24323:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 368.205746] Lustre: 24323:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 368.208036] Lustre: 24323:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 368.211805] Lustre: 24323:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 368.213816] Lustre: 24323:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 368.216696] Lustre: 24323:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 368.219572] Lustre: 24323:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 368.222431] Lustre: 24323:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 374.727812] Lustre: 24323:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 374.731825] Lustre: 24323:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 3 previous similar messages [ 374.734323] Lustre: 24323:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 374.736404] Lustre: 24323:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 374.739279] Lustre: 24323:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 374.741514] Lustre: 24323:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 374.744349] Lustre: 24323:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 374.747404] Lustre: 24323:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 374.750872] Lustre: 24323:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 374.754638] Lustre: 24323:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 374.758345] Lustre: 24323:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 374.761821] Lustre: 24323:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 384.303803] Lustre: 24351:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 384.306949] Lustre: 24351:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 5 previous similar messages [ 384.309878] Lustre: 24351:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 384.312226] Lustre: 24351:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 384.314284] Lustre: 24351:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 384.316712] Lustre: 24351:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 384.318966] Lustre: 24351:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 384.321987] Lustre: 24351:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 384.324842] Lustre: 24351:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 384.327677] Lustre: 24351:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 384.330648] Lustre: 24351:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 384.333408] Lustre: 24351:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 428.512672] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 428.512692] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 428.518898] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 431.470057] LustreError: 26535:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 431.510442] Lustre: server umount lustre-OST07c6 complete [ 434.706334] LustreError: 27036:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 434.708157] LustreError: 27036:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 434.770132] Lustre: server umount lustre-MDT0000 complete [ 440.345914] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 441.390935] Key type lgssc unregistered [ 441.513337] LNet: 27868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 441.516447] LNetError: 27868:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 441.527420] LNet: Removed LNI 192.168.201.137@tcp [ 441.806890] Key type .llcrypt unregistered [ 441.807912] Key type ._llcrypt unregistered [ 448.754293] Key type ._llcrypt registered [ 448.755138] Key type .llcrypt registered [ 448.800559] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 449.161302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 449.185907] alg: No test for adler32 (adler32-zlib) [ 450.037533] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 450.125467] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 451.703193] Key type lgssc registered [ 452.082603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 454.706936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 454.713722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 455.900514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 457.183429] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 460.248700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 460.389118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 461.819132] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 463.802857] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 465.041688] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 467.611384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 467.761720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 469.748250] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 471.529173] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 481.062094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 489.956961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 489.957353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.957892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 489.967066] LustreError: Skipped 1 previous similar message [ 490.463690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 490.466320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 490.471521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 491.490029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 491.493402] Lustre: Skipped 1 previous similar message [ 492.653134] LustreError: 31564:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 492.691712] Lustre: server umount lustre-OST0000 complete [ 496.095773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 496.097742] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 496.101131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 501.728061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 501.730352] Lustre: Skipped 2 previous similar messages [ 508.383104] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 508.417765] LustreError: 31766:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 508.420247] LustreError: 31766:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 508.455045] Lustre: server umount lustre-MDT0000 complete [ 509.789052] LustreError: 29077:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764745091 with bad export cookie 12782500423457134104 [ 509.793328] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 509.793483] LustreError: 29077:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 509.865597] LustreError: 31969:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 509.868125] LustreError: 31969:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 509.919574] Lustre: server umount lustre-MDT0001 complete [ 513.655587] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 514.777603] Key type lgssc unregistered [ 514.906342] LNet: 32500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 514.909945] LNetError: 32500:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 514.921278] LNet: Removed LNI 192.168.201.137@tcp [ 515.234088] Key type .llcrypt unregistered [ 515.235364] Key type ._llcrypt unregistered [ 522.745757] Key type ._llcrypt registered [ 522.746906] Key type .llcrypt registered [ 522.788240] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 523.214385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 523.256045] alg: No test for adler32 (adler32-zlib) [ 524.141477] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 524.247410] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 525.839250] Key type lgssc registered [ 526.279505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 529.141854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 529.147908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 530.312052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 531.785917] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 535.048833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 535.192295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 536.653471] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 538.508110] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 539.893041] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 542.433967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 542.574255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 544.533085] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 546.495827] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 554.860870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 565.217619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 565.219604] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 565.222031] Lustre: Skipped 1 previous similar message [ 565.222497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 566.894169] LustreError: 35998:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 566.937557] Lustre: server umount lustre-OST0000 complete [ 570.337677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 570.338781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 570.343215] Lustre: Skipped 1 previous similar message [ 570.345418] Lustre: Skipped 1 previous similar message [ 574.528101] LustreError: 36200:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 574.530791] LustreError: 36200:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 574.562657] Lustre: server umount lustre-MDT0000 complete [ 575.457290] LustreError: 35010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 575.464032] LustreError: 35010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 575.901717] LustreError: 33710:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764745157 with bad export cookie 16751600205812918314 [ 575.905930] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 575.906940] LustreError: 33710:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 575.974871] LustreError: 36403:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 575.977223] LustreError: 36403:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 576.035392] Lustre: server umount lustre-MDT0001 complete [ 579.508790] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 580.562818] Key type lgssc unregistered [ 580.701416] LNet: 36934:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 580.705371] LNetError: 36934:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 580.716430] LNet: Removed LNI 192.168.201.137@tcp [ 581.004403] Key type .llcrypt unregistered [ 581.006057] Key type ._llcrypt unregistered [ 589.478366] Key type ._llcrypt registered [ 589.479260] Key type .llcrypt registered [ 589.523554] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 589.923701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 589.976051] alg: No test for adler32 (adler32-zlib) [ 590.838214] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 590.931243] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 592.519214] Key type lgssc registered [ 592.893571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 595.644514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 595.650272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 596.806428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 598.213627] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 601.199424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 601.328510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 602.650126] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 604.449206] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 605.669214] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 607.940062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 608.081157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 610.043401] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 612.136736] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 623.463126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 632.288132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 632.290612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 632.294128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 633.827184] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 633.827581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 633.833694] Lustre: Skipped 3 previous similar messages [ 636.030676] LustreError: 40575:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 636.065359] Lustre: server umount lustre-MDT0000 complete [ 637.510395] LustreError: 39926:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764745219 with bad export cookie 17756743238370113399 [ 637.513673] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 637.514497] LustreError: 39926:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 637.794554] LustreError: 40463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 637.801296] LustreError: 40463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 637.801777] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [ 638.944191] LustreError: 40463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 638.944443] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 638.958257] Lustre: Skipped 1 previous similar message [ 642.914144] LustreError: 40463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 642.914212] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [ 642.922639] Lustre: Skipped 1 previous similar message [ 643.647315] LustreError: 40777:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 643.648886] LustreError: 40777:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 643.695379] Lustre: server umount lustre-MDT0001 complete [ 661.548636] LustreError: 40998:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 661.550347] LustreError: 40998:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 661.591462] Lustre: server umount lustre-OST0000 complete [ 667.107455] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 668.133312] Key type lgssc unregistered [ 668.271319] LNet: 41837:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.273730] LNetError: 41837:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 668.286300] LNet: Removed LNI 192.168.201.137@tcp [ 668.600332] Key type .llcrypt unregistered [ 668.602264] Key type ._llcrypt unregistered [ 677.818807] Key type ._llcrypt registered [ 677.819970] Key type .llcrypt registered [ 677.861565] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 678.201305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 678.244394] alg: No test for adler32 (adler32-zlib) [ 679.086256] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 679.159888] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 680.735149] Key type lgssc registered [ 681.068649] Lustre: Echo OBD driver; http://www.lustre.org/ [ 683.671284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 683.676773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 684.843807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 686.070333] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 688.964445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 689.087898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 690.308399] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 692.059771] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 693.294736] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 695.597629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 695.708424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 697.602794] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 699.416105] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 710.054063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 719.327829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 719.329827] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 719.334844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 720.235084] Lustre: lustre-OST0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 720.353181] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 720.355851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 723.307104] LustreError: 45493:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 723.340264] Lustre: server umount lustre-OST0000 complete [ 736.033422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 736.037514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 736.039405] Lustre: Skipped 3 previous similar messages [ 741.344478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 741.347644] Lustre: Skipped 1 previous similar message [ 742.273356] LustreError: 45869:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 742.276814] LustreError: 45869:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 742.309204] Lustre: server umount lustre-MDT0000 complete [ 743.622334] LustreError: 45870:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764745325 with bad export cookie 7818343653387849017 [ 743.625354] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 743.628209] LustreError: 45870:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 749.758297] LustreError: 46071:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 749.760379] LustreError: 46071:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 749.814103] Lustre: server umount lustre-MDT0001 complete [ 753.573324] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 754.749113] Key type lgssc unregistered [ 754.890488] LNet: 46602:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 754.893365] LNetError: 46602:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 754.900517] LNet: Removed LNI 192.168.201.137@tcp [ 755.241972] Key type .llcrypt unregistered [ 755.243795] Key type ._llcrypt unregistered [ 764.401919] Key type ._llcrypt registered [ 764.402879] Key type .llcrypt registered [ 764.443427] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 764.896122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 764.946779] alg: No test for adler32 (adler32-zlib) [ 765.817438] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 765.911493] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 767.503300] Key type lgssc registered [ 767.921608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 770.803470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 770.812095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 771.981832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 773.435168] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 776.384293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 776.514677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 777.885392] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 779.717465] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 780.927495] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 783.442645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 783.585426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 785.798975] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 787.681886] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 796.906086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 800.166695] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 801.424722] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 812.211260] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-12-03 2h03m12s [ 812.750750] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 817.637538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 817.637886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 817.639074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 817.641688] Lustre: Skipped 2 previous similar messages [ 818.537322] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 818.540820] Lustre: Skipped 2 previous similar messages [ 819.554052] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 819.556678] Lustre: Skipped 3 previous similar messages [ 819.714415] LustreError: 50558:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 819.743170] Lustre: server umount lustre-MDT0000 complete [ 820.981346] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 823.491721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 823.539479] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 823.643753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 823.673737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 824.679203] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 825.106900] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 826.331063] Lustre: DEBUG MARKER: Wait for df (35971) ... [ 826.874642] Lustre: DEBUG MARKER: done [ 827.871789] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 827.876044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 827.880571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 828.896359] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 828.899347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 833.902614] LustreError: 51606:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 833.904408] LustreError: 51606:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 833.944747] Lustre: server umount lustre-OST0000 complete [ 840.160499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 840.160720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 840.161432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 840.161437] Lustre: Skipped 3 previous similar messages [ 840.164974] Lustre: Skipped 1 previous similar message [ 841.340001] LustreError: 51811:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 841.342050] LustreError: 51811:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 841.363517] Lustre: server umount lustre-MDT0000 complete [ 842.566344] LustreError: 49594:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764745424 with bad export cookie 1979017343920966053 [ 842.568512] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 842.569136] LustreError: 49594:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 842.673200] Lustre: server umount lustre-MDT0001 complete [ 845.917679] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 846.942478] Key type lgssc unregistered [ 847.063399] LNet: 52543:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.066076] LNetError: 52543:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 847.075422] LNet: Removed LNI 192.168.201.137@tcp [ 847.346187] Key type .llcrypt unregistered [ 847.347782] Key type ._llcrypt unregistered [ 858.029727] Key type ._llcrypt registered [ 858.030833] Key type .llcrypt registered [ 858.074875] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 864.509642] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 864.998669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 865.009698] alg: No test for adler32 (adler32-zlib) [ 865.873555] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 865.979661] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 867.567199] Key type lgssc registered [ 867.978991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 870.895551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 873.065710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 875.070627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 877.279312] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 879.492390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 879.509916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 879.515190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 880.593899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 880.607854] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 880.643990] Lustre: lustre-MDT0000: new disk, initializing [ 880.668253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 880.675285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 882.013810] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 886.302400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 886.326109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 886.357725] Lustre: 56464:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 886.378505] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 886.380607] Lustre: Skipped 1 previous similar message [ 886.415154] Lustre: lustre-MDT0001: new disk, initializing [ 886.453284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 886.465411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 886.468829] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 887.810303] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 890.970230] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 892.154964] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 894.265138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 894.287349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 894.379205] Lustre: lustre-OST0000: new disk, initializing [ 894.381634] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 894.409585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 896.213355] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 897.144776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 897.150194] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 897.184377] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 899.537541] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 901.041456] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 901.123028] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 902.577667] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 902.658242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 905.389075] LustreError: 59404:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 905.432652] Lustre: server umount lustre-OST0000 complete [ 907.232174] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 907.236342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 907.243319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 907.744677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 907.749085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 912.864064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 912.866938] Lustre: Skipped 1 previous similar message [ 916.962829] LustreError: 59604:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 916.964591] LustreError: 59604:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 916.988878] Lustre: server umount lustre-MDT0000 complete [ 918.280246] LustreError: 56455:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764745499 with bad export cookie 13049162686325625909 [ 918.283561] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 918.284164] LustreError: 56455:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 927.715558] LustreError: 59806:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 927.718147] LustreError: 59806:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 927.770401] Lustre: server umount lustre-MDT0001 complete [ 930.737954] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 02:05:11 (1764745511) [ 933.112647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 933.269489] LustreError: 60477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 933.275408] LustreError: 60477:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 933.290649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 934.653966] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 937.514897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 937.628035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 938.902984] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 940.670564] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 941.889066] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 944.202570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 946.262955] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 948.050301] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 961.336683] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-12-03 2h05m42s [ 961.911146] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 964.464507] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 965.328682] Lustre: DEBUG MARKER: Creating a test file and stat it [ 965.342254] LustreError: 60471:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 985.456383] Lustre: lustre-MDT0000: Client f1c2aef6-64a7-4ab6-9e00-0537eadc66bc (at 192.168.201.37@tcp) reconnecting [ 1005.383181] LustreError: 60471:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1006.232220] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1007.154524] Lustre: DEBUG MARKER: done [ 1009.119854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1009.123824] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1009.128872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1010.143698] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1014.957186] LustreError: 63503:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1014.959560] LustreError: 63503:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1014.996647] Lustre: server umount lustre-OST0000 complete [ 1017.313366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1017.314571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1017.318975] Lustre: Skipped 2 previous similar messages [ 1017.321501] Lustre: Skipped 3 previous similar messages [ 1022.464768] LustreError: 63704:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1022.466917] LustreError: 63704:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1022.493404] Lustre: server umount lustre-MDT0000 complete [ 1023.392258] LustreError: 60476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1023.396615] LustreError: 60476:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1023.765343] LustreError: 62888:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764745605 with bad export cookie 13049162686325626777 [ 1023.768591] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1023.768679] LustreError: 62888:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1023.889886] Lustre: server umount lustre-MDT0001 complete [ 1027.642174] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1028.830627] Key type lgssc unregistered [ 1028.970628] LNet: 64437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.973969] LNetError: 64437:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1028.986524] LNet: Removed LNI 192.168.201.137@tcp [ 1029.311397] Key type .llcrypt unregistered [ 1029.312618] Key type ._llcrypt unregistered [ 1040.998059] Key type ._llcrypt registered [ 1040.999033] Key type .llcrypt registered [ 1041.038946] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 1047.734094] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1048.176084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1048.186649] alg: No test for adler32 (adler32-zlib) [ 1049.036072] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 1049.119047] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1050.695157] Key type lgssc registered [ 1051.066338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1053.895515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1055.848196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1057.712878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1059.703721] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1062.046540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1062.071069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1062.077173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1063.166237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1063.183569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1063.222969] Lustre: lustre-MDT0000: new disk, initializing [ 1063.254818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1063.261479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1064.612958] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1068.921726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1068.949603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1068.991528] Lustre: 68358:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1069.043357] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1069.046075] Lustre: Skipped 1 previous similar message [ 1069.089126] Lustre: lustre-MDT0001: new disk, initializing [ 1069.127803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1069.143139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1069.147969] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1070.593257] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1073.853937] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1075.204975] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1077.701553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1077.724064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1077.833670] Lustre: lustre-OST0000: new disk, initializing [ 1077.835630] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1077.869871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1079.036853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1079.039968] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1079.083498] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1079.897784] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1083.130915] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1084.598511] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1084.695875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1086.167780] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1086.248169] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1089.506863] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1089.507190] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1089.513042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1089.516976] Lustre: Skipped 1 previous similar message [ 1094.628395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1094.631484] Lustre: Skipped 1 previous similar message [ 1099.749209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1099.751879] Lustre: Skipped 1 previous similar message [ 1101.279162] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1101.298747] LustreError: 71296:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1101.344168] Lustre: server umount lustre-OST0000 complete [ 1104.865118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1104.865160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1104.866199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1104.867259] LustreError: Skipped 1 previous similar message [ 1104.870613] Lustre: Skipped 2 previous similar messages [ 1108.803636] LustreError: 71498:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1108.806589] LustreError: 71498:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1108.841642] Lustre: server umount lustre-MDT0000 complete [ 1109.984313] LustreError: 68368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1109.989093] LustreError: 68368:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1110.211301] LustreError: 68348:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764745691 with bad export cookie 9477444703519544787 [ 1110.215145] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1110.215923] LustreError: 68348:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1110.290920] LustreError: 71699:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1110.294057] LustreError: 71699:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1110.369366] Lustre: server umount lustre-MDT0001 complete [ 1113.499412] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 02:08:14 (1764745694) [ 1114.082202] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1114.666457] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 02:08:15 (1764745695) [ 1119.094742] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1120.696975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1120.837962] LustreError: 72747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1120.853597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1121.196513] Lustre: Failing over lustre-MDT0000 [ 1121.198357] LustreError: 72775:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1121.201488] LustreError: 72775:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1121.204123] LustreError: 72775:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -5 [ 1121.263954] LustreError: 72826:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1121.266433] LustreError: 72826:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1121.308947] Lustre: server umount lustre-MDT0000 complete [ 1123.607742] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 02:08:24 (1764745704) [ 1125.851869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1126.019318] LustreError: 73394:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1126.039709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1127.440537] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1130.313938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1131.798333] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1133.638200] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1134.878499] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1137.197177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1137.309449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1137.312623] Lustre: Skipped 1 previous similar message [ 1139.220622] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1140.995928] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1153.439181] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1156.065655] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1156.066383] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1156.067357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1156.067360] Lustre: Skipped 1 previous similar message [ 1156.077127] Lustre: Skipped 2 previous similar messages [ 1161.216431] LustreError: 75760:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1161.219894] LustreError: 75760:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1161.244971] Lustre: server umount lustre-MDT0000 complete [ 1162.526064] LustreError: 73376:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764745744 with bad export cookie 9477444703519545921 [ 1162.530107] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1162.533268] LustreError: 73376:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1162.635693] Lustre: server umount lustre-MDT0001 complete [ 1163.766981] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1165.361534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1168.235462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1173.471819] LustreError: 66572:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000765cd6f2 x1850469913782656/t0(0) o250->MGC192.168.201.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1173.592753] LustreError: 75001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1173.622387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1173.656683] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1175.014302] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1177.928027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1178.033320] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.515195] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1181.439853] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1182.111120] Lustre: 66574:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764745747/real 1764745747] req@00000000f7265766 x1850469913782272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764745763 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1182.742690] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1183.199161] Lustre: 66575:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764745748/real 1764745748] req@000000003aeef1d7 x1850469913782400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764745764 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1183.206626] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1186.276574] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1187.295154] Lustre: 66576:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764745752/real 1764745752] req@00000000d544ddfd x1850469913782528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764745768 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1190.879173] Lustre: 66573:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764745755/real 1764745755] req@000000006d0de4f3 x1850469913782784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764745771 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1191.393918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1191.394776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1191.398605] Lustre: Skipped 1 previous similar message [ 1191.404343] Lustre: Skipped 7 previous similar messages [ 1195.967227] LustreError: 78221:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1195.969127] LustreError: 78221:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1195.994945] Lustre: server umount lustre-MDT0000 complete [ 1196.514183] LustreError: 76818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1196.518749] LustreError: 76818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1197.308714] LustreError: 74114:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764745778 with bad export cookie 9477444703519547811 [ 1197.313120] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1197.313293] LustreError: 74114:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1197.436649] Lustre: server umount lustre-MDT0001 complete [ 1199.658328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1202.839378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1206.751545] LustreError: 66572:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000268a62c7 x1850469913803392/t0(0) o250->MGC192.168.201.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1206.873914] LustreError: 75000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1206.899455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1206.901627] Lustre: Skipped 1 previous similar message [ 1206.928399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1208.342709] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1211.297763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1211.402677] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1211.407358] Lustre: Skipped 1 previous similar message [ 1212.843097] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1214.696688] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1215.908401] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1216.492170] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1217.503164] Lustre: 66575:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764745783/real 1764745783] req@00000000473d2497 x1850469913803008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1764745799 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1221.607064] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1226.723022] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1226.723612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1226.727653] Lustre: Skipped 2 previous similar messages [ 1229.247526] LustreError: 80674:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1229.250108] LustreError: 80674:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1229.278394] Lustre: server umount lustre-MDT0000 complete [ 1230.576485] LustreError: 73375:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764745812 with bad export cookie 9477444703519549183 [ 1230.582501] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1230.695689] Lustre: server umount lustre-MDT0001 complete [ 1232.837969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1233.954468] Lustre: DEBUG MARKER: files compared the same [ 1244.894315] Lustre: server umount lustre-OST0000 complete [ 1250.545978] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1251.575594] Key type lgssc unregistered [ 1251.717405] LNet: 82304:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1251.720761] LNetError: 82304:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1251.727301] LNet: Removed LNI 192.168.201.137@tcp [ 1252.014926] Key type .llcrypt unregistered [ 1252.016743] Key type ._llcrypt unregistered [ 1261.167274] Key type ._llcrypt registered [ 1261.168691] Key type .llcrypt registered [ 1261.215515] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1261.652429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1261.672721] alg: No test for adler32 (adler32-zlib) [ 1262.528442] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 1262.625395] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1264.215151] Key type lgssc registered [ 1264.615359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1267.330242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1267.337890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1268.514606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1269.884375] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug +malloc all [ 1272.853392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1272.949613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1274.400023] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug +malloc all [ 1276.391090] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1277.760050] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1280.178158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1280.292997] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1282.310548] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug +malloc all [ 1284.430533] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1293.602297] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1298.912680] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1298.916740] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1298.923112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1298.925279] Lustre: Skipped 1 previous similar message [ 1299.884076] LustreError: 85946:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1299.913904] Lustre: server umount lustre-OST0000 complete [ 1303.420504] LustreError: 86148:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1303.422554] LustreError: 86148:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1303.450054] Lustre: server umount lustre-MDT0000 complete [ 1304.031727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1304.031796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1304.031952] LustreError: 83674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1304.031961] LustreError: 83674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1304.036233] Lustre: Skipped 2 previous similar messages [ 1304.053175] LustreError: Skipped 1 previous similar message [ 1304.922831] LustreError: 83659:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764745886 with bad export cookie 15325222713722345960 [ 1304.924829] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1304.927950] LustreError: 83659:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1304.998606] LustreError: 86350:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1305.001304] LustreError: 86350:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1305.054107] Lustre: server umount lustre-MDT0001 complete [ 1308.896939] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1310.073942] Key type lgssc unregistered [ 1310.220383] LNet: 86881:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1310.223524] LNetError: 86881:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1310.231314] LNet: Removed LNI 192.168.201.137@tcp [ 1310.560889] Key type .llcrypt unregistered [ 1310.562065] Key type ._llcrypt unregistered [ 1319.913868] Key type ._llcrypt registered [ 1319.914836] Key type .llcrypt registered [ 1319.955517] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1320.454665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1320.468874] alg: No test for adler32 (adler32-zlib) [ 1321.331889] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 1321.432401] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1323.023212] Key type lgssc registered [ 1323.399294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1326.006609] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1326.012668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1341.471289] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1346.591240] LustreError: 88228:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.137@tcp: failed processing log, type 4: rc = -110 [ 1373.215165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1375.176902] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1376.888856] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1379.373621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1379.405679] Lustre: *** cfs_fail_loc=706, val=0*** [ 1379.534380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1379.556546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1380.780277] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1383.552977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1383.670121] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1384.813351] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1386.485802] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1387.569097] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1389.024073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1389.027183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1389.032535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1390.047868] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1390.050751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1390.054175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1390.253368] LustreError: 90554:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1390.293266] Lustre: server umount lustre-OST0000 complete [ 1393.122393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1393.123358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1393.126097] Lustre: Skipped 1 previous similar message [ 1397.570484] LustreError: 90754:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1397.573690] LustreError: 90754:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1397.608097] Lustre: server umount lustre-MDT0000 complete [ 1398.240968] LustreError: 89203:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1398.250589] LustreError: 89203:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1398.959200] LustreError: 88235:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764745980 with bad export cookie 17632247391549422391 [ 1398.962276] LustreError: 88235:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1398.962687] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1399.022723] LustreError: 90956:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1399.024590] LustreError: 90956:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1399.069704] Lustre: server umount lustre-MDT0001 complete [ 1402.931811] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1403.951170] Key type lgssc unregistered [ 1404.085511] LNet: 91487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1404.087835] LNetError: 91487:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1404.094317] LNet: Removed LNI 192.168.201.137@tcp [ 1404.350506] Key type .llcrypt unregistered [ 1404.352029] Key type ._llcrypt unregistered [ 1413.003444] Key type ._llcrypt registered [ 1413.004369] Key type .llcrypt registered [ 1413.039540] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1413.340311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1413.384264] alg: No test for adler32 (adler32-zlib) [ 1414.225546] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 1414.295865] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1415.871201] Key type lgssc registered [ 1416.232499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1419.018137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1419.024527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1421.364676] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1423.979054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1424.155426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1425.539746] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1428.575341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1428.697866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1430.676646] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1433.759258] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1433.764275] Lustre: MGS: Client 9a0a5ba9-bb0e-4f57-9687-c85aa3c21564 (at 0@lo) reconnecting [ 1433.859939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1433.885156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1435.220653] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1444.320071] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1444.322100] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1444.325649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1445.346161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1445.347625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.353990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1449.133091] LustreError: 95495:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1449.169626] Lustre: server umount lustre-OST0000 complete [ 1450.464631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1450.464698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1450.465260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1450.465265] Lustre: Skipped 1 previous similar message [ 1450.474075] Lustre: Skipped 1 previous similar message [ 1455.584213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1455.586835] Lustre: Skipped 2 previous similar messages [ 1456.511876] LustreError: 95697:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1456.515577] LustreError: 95697:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1456.524995] Lustre: server umount lustre-MDT0000 complete [ 1457.879808] LustreError: 95900:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1457.881918] LustreError: 95900:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1457.934982] Lustre: server umount lustre-MDT0001 complete [ 1459.274402] Lustre: server umount MGS complete [ 1463.087408] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 02:14:03 (1764746043) [ 1472.511885] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 1475.768207] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1478.954820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1481.103109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1483.349709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1485.529179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1487.841293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1487.868089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1489.173191] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1493.416124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1493.448748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1493.537219] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1493.556241] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1493.598716] Lustre: lustre-MDT0001: new disk, initializing [ 1493.628019] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1493.636491] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1493.640965] Lustre: 100608:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1493.644599] LustreError: 100608:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1494.971723] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1499.315708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1499.349077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1499.481695] Lustre: lustre-OST0000: new disk, initializing [ 1499.484360] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1499.515842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1501.498103] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1504.639601] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1505.825223] Lustre: 101957:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000007fd9ccb x1850470296487040/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1764746103 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1507.171103] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1507.175616] Lustre: MGS: Client 435140db-a6af-426a-a166-d06f425cdb1d (at 0@lo) reconnecting [ 1507.189931] Lustre: 99893:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1507.199756] LustreError: 99893:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1507.212524] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1507.250410] Lustre: lustre-MDT0000: new disk, initializing [ 1507.314726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1508.383810] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1508.383958] Lustre: 102319:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000dd77bb72 x1850470296492288/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1764746105 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1508.394337] Lustre: 102319:0:(client.c:1637:after_reply()) Skipped 1 previous similar message [ 1508.743527] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1510.882125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1510.884919] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1513.441606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 1513.444375] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1513.445463] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1533.919928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1533.922413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1533.927846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1538.414374] LustreError: 102841:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1538.417589] LustreError: 102841:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1538.461336] Lustre: server umount lustre-OST0000 complete [ 1541.600118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1541.603060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1541.607795] Lustre: Skipped 1 previous similar message [ 1544.672839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1544.674858] Lustre: Skipped 4 previous similar messages [ 1554.399156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1554.443651] LustreError: 103043:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1554.445858] LustreError: 103043:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1554.455583] Lustre: server umount lustre-MDT0000 complete [ 1554.913177] LustreError: 100577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1554.917535] LustreError: 100577:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 1570.271221] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1570.304804] LustreError: 103247:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 1570.306802] LustreError: 103247:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1570.369566] Lustre: server umount lustre-MDT0001 complete [ 1571.693750] Lustre: server umount MGS complete [ 1575.819986] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 02:15:56 (1764746156) [ 1581.967860] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1583.048452] Key type lgssc unregistered [ 1583.175420] LNet: 104783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1583.178451] LNetError: 104783:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1583.187432] LNet: Removed LNI 192.168.201.137@tcp [ 1583.490735] Key type .llcrypt unregistered [ 1583.491769] Key type ._llcrypt unregistered [ 1590.604829] Key type ._llcrypt registered [ 1590.605848] Key type .llcrypt registered [ 1590.646808] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1591.184750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1591.218094] alg: No test for adler32 (adler32-zlib) [ 1592.066442] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 1592.154386] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1593.735121] Key type lgssc registered [ 1594.111374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1596.765242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1596.772401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1597.948590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1599.278106] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1602.212861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1602.361471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1603.634203] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1605.547984] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1606.863857] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1609.134194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1609.267134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1611.235942] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1613.142394] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1623.589085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1627.616169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1627.619061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1627.623578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1628.643574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1628.643809] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1628.644151] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1632.494097] LustreError: 108229:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1632.530831] Lustre: server umount lustre-OST0000 complete [ 1638.368369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1638.370746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1638.374669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1638.376311] Lustre: Skipped 1 previous similar message [ 1639.998307] LustreError: 108431:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1640.001633] LustreError: 108431:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1640.026014] Lustre: server umount lustre-MDT0000 complete [ 1641.289584] LustreError: 105944:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764746222 with bad export cookie 15464232202464021881 [ 1641.292434] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1641.295094] LustreError: 105944:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1641.365869] LustreError: 108633:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1641.367898] LustreError: 108633:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1641.432020] Lustre: server umount lustre-MDT0001 complete [ 1645.213348] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1646.356983] Key type lgssc unregistered [ 1646.502520] LNet: 109164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1646.505048] LNetError: 109164:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1646.513449] LNet: Removed LNI 192.168.201.137@tcp [ 1646.835590] Key type .llcrypt unregistered [ 1646.836866] Key type ._llcrypt unregistered [ 1653.677815] Key type ._llcrypt registered [ 1653.678801] Key type .llcrypt registered [ 1653.725621] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1654.059995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1654.071947] alg: No test for adler32 (adler32-zlib) [ 1654.927800] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 1655.020826] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1656.607115] Key type lgssc registered [ 1657.003092] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1659.228335] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1659.230699] LustreError: 110203:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 1659.231217] LustreError: 110204:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 1659.236048] LustreError: 110203:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 1659.239167] LustreError: 110203:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1659.243352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1659.244769] LustreError: 110203:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1660.424267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1663.133624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1663.242686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1664.670341] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1666.350575] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing lsmod [ 1667.395473] LustreError: 111380:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 1667.395485] LustreError: 111381:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 1667.402161] LustreError: 111380:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 1667.404358] LustreError: 111380:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1667.408039] LustreError: 111380:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1667.411124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1667.553082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1668.576160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1668.579864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1668.586434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1672.672214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1672.674032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1672.678278] Lustre: Skipped 1 previous similar message [ 1674.689810] LustreError: 111569:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1674.715645] Lustre: server umount lustre-MDT0000 complete [ 1676.025451] LustreError: 111571:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764746257 with bad export cookie 17269695559283486444 [ 1676.028539] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1676.030773] LustreError: 111571:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1676.767250] LustreError: 110678:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000715af5fc x1850470549174656/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1676.776863] LustreError: 111776:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1676.778614] LustreError: 111776:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1676.822337] Lustre: server umount lustre-MDT0001 complete [ 1688.307331] LustreError: 111977:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1688.310517] LustreError: 111977:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1688.351240] Lustre: server umount lustre-OST0000 complete [ 1691.240610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1691.416520] LustreError: 112454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1691.423561] LustreError: 112454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1691.445496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1692.880819] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1695.908164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1696.045392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1697.456257] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1699.297544] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1700.570609] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1702.863900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1704.913374] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1706.749193] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1716.327471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1721.826165] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1721.826930] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1721.828699] LustreError: Skipped 1 previous similar message [ 1721.831466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1721.834991] Lustre: Skipped 1 previous similar message [ 1721.836711] Lustre: Skipped 2 previous similar messages [ 1724.462125] LustreError: 114719:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1724.464193] LustreError: 114719:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1724.499800] Lustre: server umount lustre-OST0000 complete [ 1726.945909] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1726.946329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1726.953287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1726.955190] Lustre: Skipped 1 previous similar message [ 1732.064224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1732.066457] Lustre: Skipped 1 previous similar message [ 1740.255250] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1740.292163] LustreError: 114922:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1740.294776] LustreError: 114922:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1740.325060] Lustre: server umount lustre-MDT0000 complete [ 1741.625449] LustreError: 112480:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764746323 with bad export cookie 17269695559283487102 [ 1741.628836] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1741.629025] LustreError: 112480:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1741.745676] Lustre: server umount lustre-MDT0001 complete [ 1745.497193] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1746.507627] Key type lgssc unregistered [ 1746.625313] LNet: 115655:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1746.627764] LNetError: 115655:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1746.637406] LNet: Removed LNI 192.168.201.137@tcp [ 1746.947203] Key type .llcrypt unregistered [ 1746.948351] Key type ._llcrypt unregistered [ 1756.126147] Key type ._llcrypt registered [ 1756.127953] Key type .llcrypt registered [ 1756.166966] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1756.674836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1756.708573] alg: No test for adler32 (adler32-zlib) [ 1757.577431] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 1757.667696] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1759.255173] Key type lgssc registered [ 1759.676481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1762.385704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1762.392489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1763.559288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1764.947116] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1767.796811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1767.918918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1769.218717] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1771.068635] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1772.278538] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1774.599143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1774.707716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1776.518264] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1778.190166] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1789.038096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1795.851015] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1796.676759] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1798.625075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1798.628159] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1798.634181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1799.649817] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1799.650254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1799.650436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1799.658915] Lustre: Skipped 1 previous similar message [ 1803.694067] LustreError: 119396:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1803.729299] Lustre: server umount lustre-OST0000 complete [ 1808.864484] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1808.869080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1808.876936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1811.272615] LustreError: 119598:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1811.275602] LustreError: 119598:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1811.304488] Lustre: server umount lustre-MDT0000 complete [ 1812.662259] LustreError: 117751:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764746394 with bad export cookie 3679888201394251316 [ 1812.666555] LustreError: 117751:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1812.666610] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1812.731388] LustreError: 119803:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1812.733266] LustreError: 119803:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1812.789200] Lustre: server umount lustre-MDT0001 complete [ 1816.484549] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1817.562417] Key type lgssc unregistered [ 1817.686641] LNet: 120334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1817.689101] LNetError: 120334:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1817.695484] LNet: Removed LNI 192.168.201.137@tcp [ 1818.029771] Key type .llcrypt unregistered [ 1818.031542] Key type ._llcrypt unregistered [ 1825.374737] Key type ._llcrypt registered [ 1825.376396] Key type .llcrypt registered [ 1825.424527] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1825.987930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1826.013729] alg: No test for adler32 (adler32-zlib) [ 1826.892982] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 1826.975771] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1828.559312] Key type lgssc registered [ 1829.036157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1831.824353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1831.831314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1833.017063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1834.323919] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1837.402573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1837.546280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1839.096670] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1841.319851] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1842.858748] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1845.579709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1845.719993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1847.665860] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1849.571425] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1858.028966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1863.137259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1863.138400] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1863.141491] LustreError: Skipped 1 previous similar message [ 1863.144887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1863.148881] Lustre: Skipped 1 previous similar message [ 1864.686107] LustreError: 123787:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1864.724040] Lustre: server umount lustre-OST0000 complete [ 1868.034160] LustreError: 123988:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1868.037389] LustreError: 123988:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1868.071319] Lustre: server umount lustre-MDT0000 complete [ 1868.255925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1868.257126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1868.260649] LustreError: 121521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1868.263052] Lustre: Skipped 2 previous similar messages [ 1868.267773] LustreError: 121521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1869.407793] LustreError: 121502:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764746450 with bad export cookie 3537404654907676288 [ 1869.411852] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1869.414276] LustreError: 121502:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1869.492510] LustreError: 124190:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1869.494289] LustreError: 124190:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1869.541268] Lustre: server umount lustre-MDT0001 complete [ 1873.242649] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1874.254312] Key type lgssc unregistered [ 1874.389555] LNet: 124721:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1874.391887] LNetError: 124721:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1874.401378] LNet: Removed LNI 192.168.201.137@tcp [ 1874.722694] Key type .llcrypt unregistered [ 1874.724744] Key type ._llcrypt unregistered [ 1883.873129] Key type ._llcrypt registered [ 1883.874231] Key type .llcrypt registered [ 1883.921114] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1884.381121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1884.425538] alg: No test for adler32 (adler32-zlib) [ 1885.285664] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 1885.377364] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1886.967168] Key type lgssc registered [ 1887.368840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1889.955061] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1889.961180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1891.120979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1892.424733] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1895.307522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1895.428130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1896.763344] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1898.561946] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1899.827494] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1902.082654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1902.196145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1904.077691] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1906.020261] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1916.519092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 1919.210645] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1920.472442] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1920.475155] Lustre: Skipped 2 previous similar messages [ 1921.972667] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1921.974856] Lustre: Skipped 2 previous similar messages [ 1926.151313] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1926.165118] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1929.470913] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.37@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1929.474186] Lustre: Skipped 2 previous similar messages [ 1934.351752] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.37@tcp 192.168.0.[2,10]@tcp [ 1937.480926] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.37/24@tcp in log lustre-MDT0000 [ 1937.483285] Lustre: Skipped 2 previous similar messages [ 1944.570311] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.37/24@tcp [ 1944.572066] Lustre: Skipped 1 previous similar message [ 1947.618320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1947.618500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1947.619846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1947.621935] Lustre: Skipped 1 previous similar message [ 1948.780123] LustreError: 130008:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1948.813778] Lustre: server umount lustre-OST0000 complete [ 1951.712378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1951.716872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1951.724296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1951.727600] Lustre: Skipped 1 previous similar message [ 1952.135922] LustreError: 130208:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1952.138354] LustreError: 130208:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1952.171078] Lustre: server umount lustre-MDT0000 complete [ 1952.737332] LustreError: 127422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1952.737562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1952.742113] LustreError: 127422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1953.563288] LustreError: 130209:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764746535 with bad export cookie 11049415269049332778 [ 1953.568823] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1953.642896] LustreError: 130409:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1953.647062] LustreError: 130409:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1953.710875] Lustre: server umount lustre-MDT0001 complete [ 1957.337721] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 1958.400673] Key type lgssc unregistered [ 1958.528343] LNet: 130941:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1958.530511] LNetError: 130941:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1958.537287] LNet: Removed LNI 192.168.201.137@tcp [ 1958.875616] Key type .llcrypt unregistered [ 1958.876792] Key type ._llcrypt unregistered [ 1969.976898] Key type ._llcrypt registered [ 1969.978036] Key type .llcrypt registered [ 1970.058498] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 1970.644536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1970.675552] alg: No test for adler32 (adler32-zlib) [ 1971.623509] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 1971.778300] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 1973.399192] Key type lgssc registered [ 1974.171806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1979.289200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1979.304606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1980.599846] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1980.617228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1980.629784] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.37/24@tcp [ 1983.036196] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1987.929725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1988.114776] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1988.128356] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.37/24@tcp [ 1988.162730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1990.372755] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 1993.553815] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1995.611950] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1999.034372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1999.226669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2003.330347] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2006.412986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 2009.162784] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2022.882589] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2022.885609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2022.894888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2028.006130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2028.016111] Lustre: Skipped 2 previous similar messages [ 2028.994162] LustreError: 134760:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2029.080451] Lustre: server umount lustre-OST0000 complete [ 2033.123109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2033.132992] Lustre: Skipped 1 previous similar message [ 2033.140469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2033.142892] Lustre: Skipped 1 previous similar message [ 2037.638365] LustreError: 134961:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2037.641372] LustreError: 134961:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2037.688809] Lustre: server umount lustre-MDT0000 complete [ 2038.241517] LustreError: 133764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2038.248429] LustreError: 133764:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2039.406357] LustreError: 132441:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764746620 with bad export cookie 3740757595252164458 [ 2039.411106] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2039.412068] LustreError: 132441:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2039.489841] LustreError: 135162:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2039.492612] LustreError: 135162:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2039.562300] Lustre: server umount lustre-MDT0001 complete [ 2044.346333] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2045.761385] Key type lgssc unregistered [ 2045.924855] LNet: 135694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2045.929271] LNetError: 135694:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2045.940635] LNet: Removed LNI 192.168.201.137@tcp [ 2046.345468] Key type .llcrypt unregistered [ 2046.346480] Key type ._llcrypt unregistered [ 2059.056620] Key type ._llcrypt registered [ 2059.058258] Key type .llcrypt registered [ 2059.109269] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 2066.049772] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2066.444379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2066.463272] alg: No test for adler32 (adler32-zlib) [ 2067.366749] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 2067.476284] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2069.079218] Key type lgssc registered [ 2069.587883] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2074.295429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2077.653203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2080.998745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2084.481589] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2088.561107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2088.605060] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2088.616335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2089.751503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2089.779335] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2089.822911] Lustre: lustre-MDT0000: new disk, initializing [ 2089.880593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2089.894229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2091.900448] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2097.908567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2097.939547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2097.996399] Lustre: 139045:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2098.043229] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2098.046587] Lustre: Skipped 1 previous similar message [ 2098.092152] Lustre: lustre-MDT0001: new disk, initializing [ 2098.140063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2098.163244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2098.166935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2100.229461] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2104.288080] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2105.991456] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2109.229485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2109.265392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2109.414498] Lustre: lustre-OST0000: new disk, initializing [ 2109.417336] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2109.454782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2111.851775] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2114.049396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2114.055041] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2114.103756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2115.564693] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2117.280113] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2117.379466] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2119.030106] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2119.125143] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2122.029888] LustreError: 141981:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2122.071313] Lustre: server umount lustre-OST0000 complete [ 2123.743905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2123.745816] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.749924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2124.256592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2124.256709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2129.376224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2129.378996] Lustre: Skipped 2 previous similar messages [ 2133.481384] LustreError: 142181:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2133.487529] LustreError: 142181:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2133.533667] Lustre: server umount lustre-MDT0000 complete [ 2135.317376] LustreError: 140789:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764746716 with bad export cookie 1342666986850021269 [ 2135.322233] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2135.322658] LustreError: 140789:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2144.743729] LustreError: 142386:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2144.746509] LustreError: 142386:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2144.827816] Lustre: server umount lustre-MDT0001 complete [ 2148.583376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2148.848701] LustreError: 142863:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2148.861872] LustreError: 142863:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2148.897149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2150.692845] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2153.953654] LustreError: 142863:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2154.970973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2155.161323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2156.951664] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2159.563659] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2161.268764] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2164.808368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2165.008529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2167.687313] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2170.126048] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2172.111451] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2175.327946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2177.454440] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2177.581762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2182.205062] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2182.248038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2182.342525] Lustre: lustre-OST0001: new disk, initializing [ 2182.348380] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2182.410099] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2185.319903] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2188.328821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2188.335136] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2188.393485] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2189.808473] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2189.926400] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2192.127907] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2192.250742] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2194.059246] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9671caf99000.ost_server_uuid 50 [ 2194.810248] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9671caf99000.ost_server_uuid in FULL state after 0 sec [ 2203.620381] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2203.621135] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2203.629969] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2203.637516] Lustre: Skipped 1 previous similar message [ 2203.643349] Lustre: Skipped 1 previous similar message [ 2204.788787] LustreError: 147129:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2204.818026] LustreError: 147129:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2204.852469] Lustre: server umount lustre-OST0001 complete [ 2207.202134] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2207.205952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2208.740676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2208.741653] LustreError: 144459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2208.743704] Lustre: Skipped 2 previous similar messages [ 2208.750499] LustreError: 144459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2212.916322] LustreError: 147330:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2212.927142] LustreError: 147330:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2212.986307] Lustre: server umount lustre-OST0000 complete [ 2216.929189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2216.933790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2216.940258] Lustre: Skipped 1 previous similar message [ 2216.943172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2220.860158] Lustre: server umount lustre-MDT0000 complete [ 2222.846119] LustreError: 142842:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764746804 with bad export cookie 1342666986850022158 [ 2222.849539] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2222.850811] LustreError: 142842:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2222.943162] LustreError: 147733:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2222.945619] LustreError: 147733:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2223.039286] Lustre: server umount lustre-MDT0001 complete [ 2232.612895] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2234.041921] Key type lgssc unregistered [ 2234.184636] LNet: 148721:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2234.188757] LNetError: 148721:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2234.202745] LNet: Removed LNI 192.168.201.137@tcp [ 2234.614090] Key type .llcrypt unregistered [ 2234.616675] Key type ._llcrypt unregistered [ 2250.395379] Key type ._llcrypt registered [ 2250.397287] Key type .llcrypt registered [ 2250.454465] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 2258.240577] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2258.766553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2258.782645] alg: No test for adler32 (adler32-zlib) [ 2259.681835] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 2259.799896] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2261.391246] Key type lgssc registered [ 2261.860609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2265.659352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2268.420844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2271.130546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2273.986225] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2277.313556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2277.341623] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2277.355410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2278.473154] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2278.492319] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2278.535637] Lustre: lustre-MDT0000: new disk, initializing [ 2278.587878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2278.597760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2280.340434] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2285.466947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2285.503529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2285.551693] Lustre: 152642:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2285.582784] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2285.584719] Lustre: Skipped 1 previous similar message [ 2285.628411] Lustre: lustre-MDT0001: new disk, initializing [ 2285.673371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2285.690724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2285.697390] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2287.343606] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2290.934992] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2292.488714] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2295.234859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2295.267525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2295.403163] Lustre: lustre-OST0000: new disk, initializing [ 2295.406536] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2295.453554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2297.933330] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2298.120256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2298.125580] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2298.202678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2301.745311] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2303.517204] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2303.615320] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2305.455962] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2305.588768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2308.576692] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2308.579264] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2308.580138] LustreError: Skipped 1 previous similar message [ 2308.582772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2308.595678] Lustre: Skipped 1 previous similar message [ 2312.688111] LustreError: 155581:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2312.745407] Lustre: server umount lustre-OST0000 complete [ 2316.258866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2316.261922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2316.268897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2316.271382] Lustre: Skipped 1 previous similar message [ 2318.816253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2318.816821] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2320.662849] LustreError: 155782:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2320.669760] LustreError: 155782:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2320.711223] Lustre: server umount lustre-MDT0000 complete [ 2322.462588] LustreError: 152633:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764746904 with bad export cookie 13085237484627022197 [ 2322.467212] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2322.468196] LustreError: 152633:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2322.552139] LustreError: 155984:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2322.555244] LustreError: 155984:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2322.627950] Lustre: server umount lustre-MDT0001 complete [ 2326.443485] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 02:28:27 (1764746907) [ 2333.570859] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 2338.108609] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2342.979600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2346.031601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2349.115490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2352.108451] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2355.631972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2355.679409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2355.817207] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2355.860567] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2355.921769] Lustre: lustre-MDT0000: new disk, initializing [ 2355.985286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2356.001463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2358.087331] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2363.964151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2363.998843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2364.052789] Lustre: 158972:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2364.091531] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2364.094412] Lustre: Skipped 1 previous similar message [ 2364.139685] Lustre: lustre-MDT0001: new disk, initializing [ 2364.203156] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2364.230904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2364.235821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2366.247386] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2370.361836] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2372.371706] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2376.098422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2376.155651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2376.314857] Lustre: lustre-OST0000: new disk, initializing [ 2376.322353] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2376.398133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2379.359208] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2381.849182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2381.854915] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2381.928702] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2383.751532] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2386.442936] Lustre: Failing over lustre-OST0000 [ 2386.485151] LustreError: 161410:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2386.490178] LustreError: 161410:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2386.551265] Lustre: server umount lustre-OST0000 complete [ 2400.801995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2400.955421] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2400.963342] Lustre: Skipped 1 previous similar message [ 2400.976968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2400.989748] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2403.039347] Lustre: 150857:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764746968/real 1764746968] req@0000000064bcdd0d x1850471183607936/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1764746984 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2403.058205] Lustre: 150857:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2403.910495] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2405.697521] Lustre: Failing over lustre-MDT0000 [ 2405.741220] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2405.755207] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 2405.758617] Lustre: Skipped 1 previous similar message [ 2405.807679] LustreError: 162381:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2405.809948] LustreError: 162381:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2405.856993] Lustre: server umount lustre-MDT0000 complete [ 2406.373222] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2406.383497] Lustre: Skipped 1 previous similar message [ 2407.903533] Lustre: 150858:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764746973/real 1764746973] req@00000000af0c2f6d x1850471183609088/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1764746989 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2411.488511] LustreError: 158982:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2411.502979] LustreError: 158982:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2412.511281] Lustre: 150859:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1764746978/real 1764746978] req@00000000172e8b22 x1850471183610240/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1764746994 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2415.972170] LustreError: 158979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2419.960664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2420.024299] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2420.030745] Lustre: 158974:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.137@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2420.128291] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2420.193918] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2421.092530] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2421.887634] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2425.318497] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2425.329299] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2425.355267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2430.437034] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2430.438139] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2430.448105] Lustre: Skipped 1 previous similar message [ 2430.451455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2430.455089] Lustre: Skipped 1 previous similar message [ 2432.821170] LustreError: 163264:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2432.827773] LustreError: 163264:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2432.887665] Lustre: server umount lustre-OST0000 complete [ 2435.553388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2435.556749] LustreError: Skipped 1 previous similar message [ 2440.673511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2440.681156] Lustre: Skipped 3 previous similar messages [ 2448.863282] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2448.906620] LustreError: 163467:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2448.908899] LustreError: 163467:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2448.951506] Lustre: server umount lustre-MDT0000 complete [ 2450.658189] LustreError: 158964:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764747032 with bad export cookie 13085237484627024262 [ 2450.661867] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2450.664469] LustreError: 158964:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2450.811732] Lustre: server umount lustre-MDT0001 complete [ 2455.083487] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2456.336363] Key type lgssc unregistered [ 2456.486138] LNet: 164204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2456.489591] LNetError: 164204:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2456.500461] LNet: Removed LNI 192.168.201.137@tcp [ 2456.854120] Key type .llcrypt unregistered [ 2456.855923] Key type ._llcrypt unregistered [ 2479.166452] Key type ._llcrypt registered [ 2479.167905] Key type .llcrypt registered [ 2479.222225] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2479.763736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2479.853511] alg: No test for adler32 (adler32-zlib) [ 2480.753082] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 2480.856499] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2482.447866] Key type lgssc registered [ 2482.978685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2486.328804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2486.339385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2487.573064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2489.385422] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2492.937367] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2495.247085] LustreError: 165730:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2495.252712] LustreError: 165730:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2495.257179] LustreError: 165730:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2495.334211] LustreError: 166253:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2495.407313] Lustre: server umount lustre-MDT0000 complete [ 2501.845763] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 2505.853936] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2510.174374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2513.097655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2515.833519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2518.666287] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2521.475487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2521.506897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2521.613822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2521.633287] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2521.675856] Lustre: lustre-MDT0000: new disk, initializing [ 2521.715176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2521.723971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2523.406692] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2528.519167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2528.551357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2528.603725] Lustre: 168891:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2528.636645] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2528.639526] Lustre: Skipped 1 previous similar message [ 2528.682751] Lustre: lustre-MDT0001: new disk, initializing [ 2528.739678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2528.759182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2528.763509] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2530.600513] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2534.484498] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2536.047223] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2539.084501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2539.121840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2539.274392] Lustre: lustre-OST0000: new disk, initializing [ 2539.277575] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2539.324323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2541.796400] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2545.441490] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2546.695625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2546.699873] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2546.754418] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2553.034313] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2553.747145] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:32:14 (1764747134) [ 2651.615665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2651.619410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2651.624446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2654.175526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2654.175746] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2654.178304] Lustre: Skipped 1 previous similar message [ 2654.184048] Lustre: Skipped 1 previous similar message [ 2655.427765] LustreError: 172395:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2655.431140] LustreError: 172395:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2655.488215] Lustre: server umount lustre-MDT0000 complete [ 2657.169885] LustreError: 169673:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764747238 with bad export cookie 15150865656922377863 [ 2657.173490] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2657.175920] LustreError: 169673:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2659.295725] LustreError: 168897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2659.297502] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2659.305178] LustreError: 168897:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2659.309870] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2659.316197] Lustre: Skipped 1 previous similar message [ 2662.304278] LustreError: 171186:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2662.304327] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2663.298122] LustreError: 172597:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2663.300622] LustreError: 172597:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2663.363158] Lustre: server umount lustre-MDT0001 complete [ 2679.263225] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2679.280963] LustreError: 172798:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2679.284244] LustreError: 172798:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2679.334131] Lustre: server umount lustre-OST0000 complete [ 2683.223039] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 2686.984408] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2691.740396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2694.556200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2697.176285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2699.811038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2702.771645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2702.808396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2702.917710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2702.935661] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2702.976353] Lustre: lustre-MDT0000: new disk, initializing [ 2703.027512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2703.038191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2704.898230] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2710.316725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2710.351728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2710.396424] Lustre: 175132:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2710.443796] LustreError: 175144:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2710.452160] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2710.455195] Lustre: Skipped 1 previous similar message [ 2710.497264] Lustre: lustre-MDT0001: new disk, initializing [ 2710.576208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2710.580191] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2712.534479] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2716.765527] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2718.519547] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2721.665146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2721.702682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2721.815591] Lustre: lustre-OST0000: new disk, initializing [ 2721.818850] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2721.872040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2721.876492] Lustre: Skipped 1 previous similar message [ 2724.759798] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2728.574675] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2730.335709] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2730.502814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2730.508498] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2730.552991] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2731.469501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2733.228271] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2733.324657] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2735.585890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2735.586330] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2735.591799] LustreError: Skipped 1 previous similar message [ 2735.594985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2735.600145] Lustre: Skipped 1 previous similar message [ 2736.253168] LustreError: 178078:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2736.257931] LustreError: 178078:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2736.320641] Lustre: server umount lustre-OST0000 complete [ 2740.705192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2740.710788] Lustre: Skipped 1 previous similar message [ 2744.154138] Lustre: server umount lustre-MDT0000 complete [ 2745.683452] LustreError: 176880:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764747327 with bad export cookie 15150865656922536679 [ 2745.686639] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2745.688231] LustreError: 176880:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2745.760227] LustreError: 178482:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2745.763095] LustreError: 178482:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2745.821739] Lustre: server umount lustre-MDT0001 complete [ 2749.206652] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:35:29 (1764747329) [ 2755.231944] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 2758.908352] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2762.586498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2765.239321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2767.854720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2770.472951] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2773.495435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2773.528895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2773.642046] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2773.659252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2773.698397] Lustre: lustre-MDT0000: new disk, initializing [ 2773.729681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2773.737614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2775.395850] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2780.477587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2780.511093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2780.572531] Lustre: 181468:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2780.702541] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2782.380360] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2786.052812] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2787.645355] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2790.684677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2790.715317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2793.337730] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2796.935178] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2797.060615] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2797.111621] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2802.148408] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2802.150078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2802.151194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2802.151203] Lustre: Skipped 3 previous similar messages [ 2802.152960] LustreError: Skipped 1 previous similar message [ 2802.158189] Lustre: Skipped 1 previous similar message [ 2805.491076] LustreError: 184015:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2805.496802] LustreError: 184015:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2805.548375] Lustre: server umount lustre-OST0000 complete [ 2813.304376] Lustre: server umount lustre-MDT0000 complete [ 2814.975290] LustreError: 183224:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764747396 with bad export cookie 15150865656922537561 [ 2814.978929] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2814.980841] LustreError: 183224:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2815.126264] Lustre: server umount lustre-MDT0001 complete [ 2818.741586] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:36:39 (1764747399) [ 2825.337301] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 2828.975510] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2832.993574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2835.643247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2838.288621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2841.283387] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2844.238215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2844.273139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2844.378232] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2844.399771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2844.402406] Lustre: Skipped 3 previous similar messages [ 2844.442985] Lustre: lustre-MDT0000: new disk, initializing [ 2844.445185] Lustre: Skipped 2 previous similar messages [ 2844.498982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2844.502052] Lustre: Skipped 2 previous similar messages [ 2844.510741] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2844.515666] Lustre: Skipped 2 previous similar messages [ 2846.122166] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2850.966042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2851.000279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2851.049418] Lustre: 187405:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2851.170589] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2852.740214] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2856.272412] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2857.782736] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2860.781305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2860.828585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2862.653445] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2863.423928] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2866.792961] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2870.893090] LustreError: 189950:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2870.895236] LustreError: 189950:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2870.938756] Lustre: server umount lustre-OST0000 complete [ 2872.800730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2872.802043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2872.806026] Lustre: Skipped 3 previous similar messages [ 2872.809164] Lustre: Skipped 5 previous similar messages [ 2882.048220] Lustre: server umount lustre-MDT0000 complete [ 2883.040183] LustreError: 187415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2883.047898] LustreError: 187415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2883.531055] LustreError: 189159:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764747465 with bad export cookie 15150865656922538625 [ 2883.536377] LustreError: 189159:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2883.536509] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2893.853398] Lustre: server umount lustre-MDT0001 complete [ 2898.084274] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2899.315290] Key type lgssc unregistered [ 2899.456451] LNet: 190886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2899.459966] LNetError: 190886:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2899.466485] LNet: Removed LNI 192.168.201.137@tcp [ 2899.826404] Key type .llcrypt unregistered [ 2899.828063] Key type ._llcrypt unregistered [ 2910.630904] Key type ._llcrypt registered [ 2910.632831] Key type .llcrypt registered [ 2910.674973] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2911.047593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2911.093549] alg: No test for adler32 (adler32-zlib) [ 2911.970987] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 2912.071995] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2913.663410] Key type lgssc registered [ 2914.089497] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2917.070277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2917.078614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2918.257984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2919.677601] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2922.809420] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2922.969112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2924.463229] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2926.577302] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2928.050753] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2930.808699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2930.922685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2932.854484] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 2934.766466] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2943.207076] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2958.820355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2958.821046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2958.828026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2958.830853] Lustre: Skipped 1 previous similar message [ 2963.182112] LustreError: 194530:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2963.218816] Lustre: server umount lustre-OST0000 complete [ 2969.056856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2969.056894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2969.057486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2969.061631] Lustre: Skipped 2 previous similar messages [ 2969.064826] LustreError: Skipped 1 previous similar message [ 2970.559832] LustreError: 194732:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2970.562596] LustreError: 194732:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2970.595104] Lustre: server umount lustre-MDT0000 complete [ 2972.055684] LustreError: 193880:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764747553 with bad export cookie 7314468114751114300 [ 2972.058680] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2972.060556] LustreError: 193880:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2972.136627] LustreError: 194934:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2972.139181] LustreError: 194934:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2972.199955] Lustre: server umount lustre-MDT0001 complete [ 2975.905187] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 2976.991821] Key type lgssc unregistered [ 2977.122426] LNet: 195464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2977.125405] LNetError: 195464:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2977.133320] LNet: Removed LNI 192.168.201.137@tcp [ 2977.453563] Key type .llcrypt unregistered [ 2977.454947] Key type ._llcrypt unregistered [ 2988.666994] Key type ._llcrypt registered [ 2988.669142] Key type .llcrypt registered [ 2988.735400] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 2989.223761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2989.384883] alg: No test for adler32 (adler32-zlib) [ 2990.380845] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 2990.543660] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 2992.199153] Key type lgssc registered [ 2992.879781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2997.570557] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2997.585029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2998.862815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3001.489368] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3005.633567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3005.787508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3007.323548] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3009.409566] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3010.908610] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3013.780602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3013.939805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3016.186759] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3018.353811] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3024.166354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3029.474737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3029.474793] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3029.475757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3029.480423] Lustre: Skipped 1 previous similar message [ 3034.594040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3034.596850] Lustre: Skipped 3 previous similar messages [ 3039.715139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3039.718043] Lustre: Skipped 2 previous similar messages [ 3041.247105] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3041.265161] LustreError: 199086:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3041.304722] Lustre: server umount lustre-OST0000 complete [ 3043.422618] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9671c6ff9000.ost_server_uuid 50 [ 3044.034232] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9671c6ff9000.ost_server_uuid in DISCONN state after 0 sec [ 3044.666481] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3055.014900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3055.015870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3055.020563] Lustre: Skipped 1 previous similar message [ 3055.023574] Lustre: Skipped 1 previous similar message [ 3060.192574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3060.195818] Lustre: Skipped 2 previous similar messages [ 3061.123490] LustreError: 199485:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3061.126724] LustreError: 199485:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3061.167125] Lustre: server umount lustre-MDT0000 complete [ 3062.716448] LustreError: 196820:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764747644 with bad export cookie 12679626259589771402 [ 3062.719251] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3062.722077] LustreError: 196820:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3062.796850] LustreError: 199687:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3062.799243] LustreError: 199687:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3062.862094] Lustre: server umount lustre-MDT0001 complete [ 3066.316579] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:40:47 (1764747647) [ 3068.789442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3068.982646] LustreError: 200358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3068.989770] LustreError: 200358:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3069.010078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3070.532220] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3074.017538] LustreError: 200358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3074.022909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3074.024891] LustreError: 200358:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3074.179380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3075.667598] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3077.713387] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3079.013633] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3081.607347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3083.974720] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3086.189950] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3088.869436] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3088.899622] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3088.968478] Lustre: lustre-OST0001: new disk, initializing [ 3088.971727] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3089.003728] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3089.006541] Lustre: Skipped 1 previous similar message [ 3091.011701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3091.016574] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3091.048136] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3091.133561] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3094.120478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3094.666964] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3099.108067] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3099.112358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3099.119183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3101.808078] LustreError: 203560:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3101.810354] LustreError: 203560:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3101.848767] Lustre: server umount lustre-OST0000 complete [ 3104.110577] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3104.197904] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3104.738356] LustreError: 201952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3104.746237] LustreError: 201952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3105.837816] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3105.943351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3109.857544] LustreError: 201952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3109.863403] LustreError: 201952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3114.978195] LustreError: 204175:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3114.984460] LustreError: 204175:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3116.511880] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3116.515739] LustreError: Skipped 1 previous similar message [ 3116.518193] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3116.524343] Lustre: Skipped 1 previous similar message [ 3116.526587] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3116.529254] Lustre: Skipped 2 previous similar messages [ 3122.096115] LustreError: 204262:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3122.098171] LustreError: 204262:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3122.142183] Lustre: server umount lustre-OST0001 complete [ 3125.217034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3125.220871] Lustre: Skipped 2 previous similar messages [ 3129.570939] Lustre: server umount lustre-MDT0000 complete [ 3130.336490] LustreError: 200358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3130.343367] LustreError: 200358:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3130.943373] LustreError: 200339:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764747712 with bad export cookie 12679626259589772473 [ 3130.946066] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3130.947913] LustreError: 200339:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3131.016735] LustreError: 204667:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3131.019139] LustreError: 204667:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3131.079249] Lustre: server umount lustre-MDT0001 complete [ 3133.185488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3134.509405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3135.967433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3137.442491] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3139.859600] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:42:00 (1764747720) [ 3142.372824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3142.524551] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3142.540474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3142.629247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3144.169224] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3147.497704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3147.550026] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3147.568290] Lustre: 206155:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3149.240075] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3151.449217] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3152.955337] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3155.871811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3155.960320] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3156.756323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 3158.525955] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3160.994317] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3163.989973] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3164.020900] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3166.274517] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3168.301694] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3172.320289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3172.320426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3172.323390] LustreError: Skipped 2 previous similar messages [ 3172.323962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3172.328830] Lustre: Skipped 1 previous similar message [ 3172.330424] Lustre: Skipped 5 previous similar messages [ 3172.880701] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3175.341124] LustreError: 209341:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3175.346437] LustreError: 209341:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3175.380412] Lustre: server umount lustre-OST0000 complete [ 3177.955667] LustreError: 207783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3177.962975] LustreError: 207783:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3188.195972] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3192.276706] Lustre: server umount lustre-OST0001 complete [ 3193.823769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3193.827080] Lustre: Skipped 2 previous similar messages [ 3199.796456] Lustre: server umount lustre-MDT0000 complete [ 3201.190311] LustreError: 207803:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764747782 with bad export cookie 12679626259589773859 [ 3201.194156] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3201.195393] LustreError: 207803:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3201.303687] Lustre: server umount lustre-MDT0001 complete [ 3203.285936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3204.724921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3206.042434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3207.335537] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3209.489583] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:43:10 (1764747790) [ 3214.603284] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 3217.929445] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 3221.505884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3223.655021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3225.620277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3227.695154] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3229.826928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3229.850787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3229.934553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3229.947497] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3229.984070] Lustre: lustre-MDT0000: new disk, initializing [ 3230.011492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3230.014047] Lustre: Skipped 3 previous similar messages [ 3230.019595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3231.310682] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3235.559182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3235.593273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3235.631267] Lustre: 213773:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3235.668902] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3235.671541] Lustre: Skipped 1 previous similar message [ 3235.711356] Lustre: lustre-MDT0001: new disk, initializing [ 3235.763796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3235.768192] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3237.255418] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3240.550890] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3241.765708] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3244.079710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3244.103975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3244.199202] Lustre: lustre-OST0000: new disk, initializing [ 3244.201626] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3246.159147] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3249.213184] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3250.553574] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3253.750272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3253.753808] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3253.782049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3254.704340] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 3255.981346] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3256.054507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3258.849616] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3258.851952] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3258.853309] LustreError: Skipped 3 previous similar messages [ 3258.854118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3258.859305] Lustre: Skipped 2 previous similar messages [ 3258.864573] Lustre: Skipped 9 previous similar messages [ 3262.765161] LustreError: 216740:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3262.766866] LustreError: 216740:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3262.801965] Lustre: server umount lustre-OST0000 complete [ 3270.229022] Lustre: server umount lustre-MDT0000 complete [ 3271.495118] LustreError: 214550:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764747853 with bad export cookie 12679626259589775602 [ 3271.497703] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3271.499699] LustreError: 214550:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3271.629173] Lustre: server umount lustre-MDT0001 complete [ 3274.382496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3274.525942] LustreError: 217618:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3274.532664] LustreError: 217618:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3275.899377] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3278.783811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3280.193304] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3281.922027] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3283.105388] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3285.400639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3287.344284] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3289.062113] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3290.393177] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3300.653442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3301.923803] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3301.992301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3303.903767] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3303.906354] LustreError: Skipped 1 previous similar message [ 3308.756654] Lustre: server umount lustre-OST0000 complete [ 3310.710152] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3312.070396] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3335.730126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3335.831679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3335.834969] Lustre: Skipped 5 previous similar messages [ 3337.849333] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3339.707970] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3340.229349] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3343.277671] LustreError: 222061:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3343.280586] LustreError: 222061:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3343.315996] Lustre: server umount lustre-OST0000 complete [ 3344.595935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3344.600192] Lustre: Skipped 5 previous similar messages [ 3346.705413] Lustre: server umount lustre-MDT0000 complete [ 3347.950412] LustreError: 221394:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764747929 with bad export cookie 12679626259589776484 [ 3347.953693] LustreError: 221394:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3347.954657] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3348.080128] Lustre: server umount lustre-MDT0001 complete [ 3350.958199] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:45:31 (1764747931) [ 3353.271791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3353.421239] LustreError: 223139:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3354.868376] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3357.969167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3359.432538] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3361.317380] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3362.564261] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3364.902410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3366.889647] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3368.695781] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3370.214647] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3378.458309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 3379.790758] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3379.867365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3382.261461] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3382.287419] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3382.339985] Lustre: lustre-OST0001: new disk, initializing [ 3382.342197] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3384.353321] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3387.500937] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3388.890519] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3389.438222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3389.442492] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3389.474275] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3389.986347] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3391.347164] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3391.430828] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3394.530896] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3394.532151] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3394.533774] LustreError: Skipped 2 previous similar messages [ 3394.538392] Lustre: Skipped 9 previous similar messages [ 3398.207054] Lustre: server umount lustre-OST0001 complete [ 3400.020976] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3400.083205] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3401.295830] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3401.364610] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3425.092935] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3427.183537] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3429.095731] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3429.665088] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3436.406413] Lustre: server umount lustre-OST0001 complete [ 3454.417587] Lustre: server umount lustre-OST0000 complete [ 3460.576073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3460.578094] LustreError: Skipped 2 previous similar messages [ 3461.849822] Lustre: server umount lustre-MDT0000 complete [ 3463.053424] LustreError: 223119:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764748044 with bad export cookie 12679626259589777415 [ 3463.059759] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3469.231609] Lustre: server umount lustre-MDT0001 complete [ 3471.037979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3472.220448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3473.436554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3474.808418] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3477.091815] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 02:47:37 (1764748057) [ 3479.285053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3479.368369] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3479.379888] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3479.446882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3479.449616] Lustre: Skipped 5 previous similar messages [ 3480.711429] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3483.604558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3483.638601] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3483.648786] Lustre: 231174:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3484.978275] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3486.608659] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3487.666464] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3489.710678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3489.771335] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3491.498304] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3493.257388] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3502.624890] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3502.650140] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3504.646916] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3506.516491] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3508.024636] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 3509.221560] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3509.296163] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3510.465202] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3510.534651] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3511.779217] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9671e0251000.ost_server_uuid 50 [ 3512.295942] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9671e0251000.ost_server_uuid in FULL state after 0 sec [ 3512.653777] Lustre: Permanently deactivating lustre-OST0001 [ 3512.658968] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3513.468745] Lustre: Permanently reactivating lustre-OST0001 [ 3515.368486] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3515.374718] Lustre: Skipped 7 previous similar messages [ 3520.557114] LustreError: 235155:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3520.559756] LustreError: 235155:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 3520.579670] Lustre: server umount lustre-OST0001 complete [ 3525.607107] LustreError: 232796:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3525.616094] LustreError: 232796:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 23 previous similar messages [ 3536.351106] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3536.402400] Lustre: server umount lustre-OST0000 complete [ 3543.766656] Lustre: server umount lustre-MDT0000 complete [ 3544.983185] LustreError: 233713:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764748126 with bad export cookie 12679626259589778675 [ 3544.985336] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3544.986510] LustreError: 233713:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 3545.102861] Lustre: server umount lustre-MDT0001 complete [ 3546.815170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3547.967750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3549.125068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3550.352169] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3552.320952] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 02:48:53 (1764748133) [ 3552.622623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3554.683041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3554.789643] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3554.802352] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3554.805267] Lustre: Skipped 5 previous similar messages [ 3556.135807] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3558.802324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3558.827583] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3558.838402] Lustre: 237311:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3560.104063] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3561.816856] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3562.868655] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3565.019415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3565.045858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3565.111203] Lustre: Permanently deactivating lustre-OST0000 [ 3565.115476] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3567.001100] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3568.886304] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3571.178490] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3572.218274] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3572.268095] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3573.245766] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3575.023049] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3575.592355] Lustre: Permanently reactivating lustre-OST0000 [ 3581.945961] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3581.973214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3589.694147] Lustre: server umount lustre-OST0001 complete [ 3592.159841] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3592.161644] LustreError: Skipped 8 previous similar messages [ 3597.010736] Lustre: server umount lustre-OST0000 complete [ 3604.513806] Lustre: server umount lustre-MDT0000 complete [ 3605.913409] LustreError: 237304:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764748187 with bad export cookie 12679626259589780313 [ 3605.916924] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3605.920339] LustreError: 237304:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3606.064085] Lustre: server umount lustre-MDT0001 complete [ 3609.519384] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 3610.459947] Key type lgssc unregistered [ 3610.583308] LNet: 241708:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3610.585604] LNetError: 241708:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3610.596293] LNet: Removed LNI 192.168.201.137@tcp [ 3610.877481] Key type .llcrypt unregistered [ 3610.878387] Key type ._llcrypt unregistered [ 3620.229048] Key type ._llcrypt registered [ 3620.230829] Key type .llcrypt registered [ 3620.272885] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 3620.752101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3620.761654] alg: No test for adler32 (adler32-zlib) [ 3621.635068] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 3621.738927] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 3623.327162] Key type lgssc registered [ 3623.709874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3624.953744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3627.343813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3627.349916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3628.528483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3629.914358] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3632.874692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3632.898256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3632.930691] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3632.940314] Lustre: 243121:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3632.946226] Lustre: Permanently deactivating lustre-MDT0001 [ 3632.951306] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3633.054691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3634.297131] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3635.980516] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3636.520812] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3636.523231] LustreError: 243164:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3636.526858] LustreError: 243164:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3636.530754] LustreError: 243164:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 1, failed: rc = -108 [ 3637.037205] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3639.309191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3639.454727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3641.425648] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3642.532669] LustreError: 244753:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3642.540418] LustreError: 244753:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3643.213572] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3645.657884] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3645.729645] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3647.711389] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3649.657757] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3650.545508] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3653.612124] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3653.612137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3655.468137] Lustre: Permanently reactivating lustre-MDT0001 [ 3655.475746] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3655.479679] Lustre: Skipped 2 previous similar messages [ 3658.665780] Lustre: lustre-MDT0001: Client 8b31ec82-699c-49f0-ac1a-95497b3741aa (at 192.168.201.37@tcp) reconnecting [ 3660.890362] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3661.523980] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3663.216673] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3663.295341] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3664.553702] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3664.626216] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3665.424750] Lustre: Permanently deactivating lustre-MDT0001 [ 3665.429350] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3665.432228] Lustre: Skipped 2 previous similar messages [ 3668.606700] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3674.080215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3674.082736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3674.086776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3676.130841] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3676.133338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3676.136103] Lustre: Skipped 2 previous similar messages [ 3676.139545] Lustre: Skipped 2 previous similar messages [ 3679.422725] LustreError: 247249:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3679.453133] Lustre: server umount lustre-MDT0000 complete [ 3680.823129] LustreError: 243115:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764748262 with bad export cookie 15876198333787966013 [ 3680.825534] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3680.827321] LustreError: 243115:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3680.903676] LustreError: 247451:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3680.905849] LustreError: 247451:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3680.961601] Lustre: server umount lustre-MDT0001 complete [ 3688.365112] LustreError: 247653:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3688.367150] LustreError: 247653:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3688.382159] Lustre: server umount lustre-OST0000 complete [ 3692.300260] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 02:51:13 (1764748273) [ 3695.919194] LustreError: 248658:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3695.921283] LustreError: 248658:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3695.967163] Lustre: server umount lustre-OST0001 complete [ 3698.034557] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 3701.453516] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 3704.927672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3707.018526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3708.962130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3710.982228] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3713.241865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3713.267287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3713.345978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3713.349344] Lustre: Skipped 2 previous similar messages [ 3713.359313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3713.395284] Lustre: lustre-MDT0000: new disk, initializing [ 3713.418225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3713.424218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3714.602048] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3718.495953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3718.516315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3718.543423] Lustre: 250838:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3718.567831] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3718.569375] Lustre: Skipped 1 previous similar message [ 3718.605530] Lustre: lustre-MDT0001: new disk, initializing [ 3718.646097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3718.650043] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3719.895673] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3723.006984] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3724.166188] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3726.440335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3726.464095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3726.568702] Lustre: lustre-OST0000: new disk, initializing [ 3726.570879] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3726.599229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3726.601077] Lustre: Skipped 1 previous similar message [ 3727.619119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3727.622575] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3727.655218] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3728.531530] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3731.807738] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3732.525460] LustreError: 252756:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3736.781052] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3736.805570] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3736.858355] Lustre: lustre-OST0001: new disk, initializing [ 3736.860078] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3738.794215] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3741.822168] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3742.348394] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3742.615098] LustreError: 252756:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3742.622309] LustreError: 252756:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3744.764800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3744.769295] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3744.795547] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3752.711087] LustreError: 252756:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3754.977447] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3754.979084] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3754.979430] LustreError: Skipped 1 previous similar message [ 3754.983896] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3754.986701] Lustre: Skipped 2 previous similar messages [ 3759.662749] LustreError: 254347:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3759.665168] LustreError: 254347:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3759.685103] Lustre: server umount lustre-OST0001 complete [ 3760.103072] LustreError: 252761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3760.107533] LustreError: 252761:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3765.218976] LustreError: 254240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3765.224461] LustreError: 254240:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3770.340024] LustreError: 252564:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3770.345579] LustreError: 252564:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3773.920469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3773.920556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3773.923426] LustreError: Skipped 1 previous similar message [ 3773.926950] Lustre: Skipped 1 previous similar message [ 3773.927582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3773.931197] Lustre: Skipped 1 previous similar message [ 3775.459497] LustreError: 252761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3775.467907] LustreError: 252761:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3777.775106] LustreError: 254570:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3777.778962] LustreError: 254570:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3777.821292] Lustre: server umount lustre-OST0000 complete [ 3780.064091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3780.068639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3780.073728] Lustre: Skipped 1 previous similar message [ 3780.075477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3780.077712] Lustre: Skipped 2 previous similar messages [ 3785.494671] Lustre: server umount lustre-MDT0000 complete [ 3785.695985] LustreError: 250847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3785.701704] LustreError: 250847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3786.800289] LustreError: 250830:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764748368 with bad export cookie 15876198333787968610 [ 3786.803494] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3786.806874] LustreError: 250830:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3801.055118] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3801.093612] LustreError: 254974:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3801.095777] LustreError: 254974:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3801.150458] Lustre: server umount lustre-MDT0001 complete [ 3804.940103] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 3806.130340] Key type lgssc unregistered [ 3806.266410] LNet: 255507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3806.270705] LNetError: 255507:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3806.279449] LNet: Removed LNI 192.168.201.137@tcp [ 3806.631342] Key type .llcrypt unregistered [ 3806.632524] Key type ._llcrypt unregistered [ 3818.280589] Key type ._llcrypt registered [ 3818.281692] Key type .llcrypt registered [ 3818.320706] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 3824.735847] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 3825.211739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3825.248518] alg: No test for adler32 (adler32-zlib) [ 3826.103437] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 3826.188234] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 3827.775185] Key type lgssc registered [ 3828.156768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3831.152078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3833.360188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3835.453417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3837.726890] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3840.104983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3840.123337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3840.129665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3841.212710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3841.224857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3841.263344] Lustre: lustre-MDT0000: new disk, initializing [ 3841.288810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3841.298415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3842.541522] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3846.750347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3846.773461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3846.812468] Lustre: 259429:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3846.841521] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3846.843280] Lustre: Skipped 1 previous similar message [ 3846.878263] Lustre: lustre-MDT0001: new disk, initializing [ 3846.910879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3846.928099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3846.932932] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3848.395446] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3851.695165] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3852.981113] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3855.215971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3855.237936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3855.339105] Lustre: lustre-OST0000: new disk, initializing [ 3855.341228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3855.372280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3857.163317] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3860.151354] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3861.499961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3861.504499] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3861.505496] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3861.549125] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3861.606816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3862.885761] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3862.962954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3866.593424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3866.594382] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3866.595660] LustreError: Skipped 1 previous similar message [ 3866.599018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3866.600956] Lustre: Skipped 1 previous similar message [ 3869.678666] LustreError: 262367:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3869.720501] Lustre: server umount lustre-OST0000 complete [ 3871.712482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3871.713130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3871.715863] Lustre: Skipped 2 previous similar messages [ 3876.832801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3876.834593] Lustre: Skipped 2 previous similar messages [ 3877.122516] LustreError: 262568:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3877.125417] LustreError: 262568:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3877.160697] Lustre: server umount lustre-MDT0000 complete [ 3878.408504] LustreError: 262570:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764748459 with bad export cookie 12444622302584184028 [ 3878.412988] LustreError: 262570:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3878.413075] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3878.477913] LustreError: 262771:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3878.480025] LustreError: 262771:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3878.537877] Lustre: server umount lustre-MDT0001 complete [ 3881.444932] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 02:54:22 (1764748462) [ 3883.746464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3883.941230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3885.260124] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3888.155211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3888.290788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3889.687142] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3891.515129] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3892.803720] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3895.189328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3897.213791] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3898.996275] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3954.655915] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3954.658904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3954.663636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3955.680808] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3958.831151] LustreError: 265786:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3958.834620] LustreError: 265786:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3958.892887] Lustre: server umount lustre-OST0000 complete [ 3960.442328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3964.616100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3964.735165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3964.738058] Lustre: Skipped 1 previous similar message [ 3966.638705] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 3968.440977] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3976.037872] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3976.043575] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3981.161541] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3981.165492] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4006.881517] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4006.882409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4006.884512] LustreError: Skipped 1 previous similar message [ 4006.885106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4006.888959] Lustre: Skipped 2 previous similar messages [ 4006.890414] Lustre: Skipped 2 previous similar messages [ 4007.981362] LustreError: 267450:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4007.983410] LustreError: 267450:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4008.030265] Lustre: server umount lustre-OST0000 complete [ 4011.488033] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4011.490140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4015.454322] Lustre: server umount lustre-MDT0000 complete [ 4016.765259] LustreError: 263423:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764748598 with bad export cookie 12444622302584184896 [ 4016.770172] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4016.770470] LustreError: 263423:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4016.847031] LustreError: 267853:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4016.849612] LustreError: 267853:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4016.910574] Lustre: server umount lustre-MDT0001 complete [ 4020.844697] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 4021.843735] Key type lgssc unregistered [ 4021.977365] LNet: 268385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4021.980267] LNetError: 268385:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4021.990313] LNet: Removed LNI 192.168.201.137@tcp [ 4022.276734] Key type .llcrypt unregistered [ 4022.278651] Key type ._llcrypt unregistered [ 4031.551446] Key type ._llcrypt registered [ 4031.552392] Key type .llcrypt registered [ 4031.591580] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4031.921509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4031.951060] alg: No test for adler32 (adler32-zlib) [ 4032.808679] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 4032.887279] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 4034.471144] Key type lgssc registered [ 4034.819118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4037.483412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4037.489974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4038.681765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4040.190591] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4043.299500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4043.420257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4044.744338] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4046.715711] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4048.519249] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4053.061488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4053.334617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4058.990956] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4063.661907] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4064.771357] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4088.289981] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4088.290729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4088.291916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4088.301140] Lustre: Skipped 1 previous similar message [ 4091.004785] LustreError: 272778:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4091.069511] Lustre: server umount lustre-OST0000 complete [ 4093.408912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4093.410210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4093.415379] Lustre: Skipped 1 previous similar message [ 4093.419061] Lustre: Skipped 2 previous similar messages [ 4098.528479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4098.531302] Lustre: Skipped 1 previous similar message [ 4103.648067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4103.650922] Lustre: Skipped 1 previous similar message [ 4107.231140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4107.269298] LustreError: 272979:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4107.272531] LustreError: 272979:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4107.309281] Lustre: server umount lustre-MDT0000 complete [ 4107.744453] LustreError: 269761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4107.751956] LustreError: 269761:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4108.744322] LustreError: 270478:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764748690 with bad export cookie 11301548355614536685 [ 4108.747510] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4108.748804] LustreError: 270478:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4108.828305] LustreError: 273182:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4108.830887] LustreError: 273182:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4108.887909] Lustre: server umount lustre-MDT0001 complete [ 4112.999281] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 4114.220630] Key type lgssc unregistered [ 4114.341346] LNet: 273713:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4114.343846] LNetError: 273713:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4114.351350] LNet: Removed LNI 192.168.201.137@tcp [ 4114.667813] Key type .llcrypt unregistered [ 4114.669092] Key type ._llcrypt unregistered [ 4122.566664] Key type ._llcrypt registered [ 4122.567793] Key type .llcrypt registered [ 4122.614579] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4122.957312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4122.990668] alg: No test for adler32 (adler32-zlib) [ 4123.862826] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 4123.959849] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 4125.551462] Key type lgssc registered [ 4125.999645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4129.380661] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4129.390639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4130.626784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4132.193933] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4135.562571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4135.722576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4137.527339] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4139.708568] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4141.320123] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4143.920916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4144.065464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4146.326623] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4148.439240] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4156.400527] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4166.626620] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4166.628844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4166.629677] LustreError: Skipped 1 previous similar message [ 4166.635037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4166.637315] Lustre: Skipped 1 previous similar message [ 4168.692195] LustreError: 277319:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4168.731752] Lustre: server umount lustre-OST0000 complete [ 4171.746080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4171.746268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4171.747190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4171.750247] Lustre: Skipped 2 previous similar messages [ 4176.131049] LustreError: 277520:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4176.133663] LustreError: 277520:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4176.166786] Lustre: server umount lustre-MDT0000 complete [ 4176.863961] LustreError: 274905:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4176.871960] LustreError: 274905:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4177.600410] LustreError: 274888:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764748759 with bad export cookie 13449665919199852309 [ 4177.602684] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4177.606292] LustreError: 274888:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4177.680385] LustreError: 277721:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4177.683528] LustreError: 277721:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4177.755078] Lustre: server umount lustre-MDT0001 complete [ 4181.819981] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 4183.028489] Key type lgssc unregistered [ 4183.170392] LNet: 278252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4183.173262] LNetError: 278252:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4183.184387] LNet: Removed LNI 192.168.201.137@tcp [ 4183.505888] Key type .llcrypt unregistered [ 4183.507622] Key type ._llcrypt unregistered [ 4194.115547] Key type ._llcrypt registered [ 4194.117129] Key type .llcrypt registered [ 4194.163105] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4194.638482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4194.647861] alg: No test for adler32 (adler32-zlib) [ 4195.539549] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 4195.650089] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 4197.239156] Key type lgssc registered [ 4197.671882] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4200.775255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4200.785527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4201.985157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4203.584309] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4207.169848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4207.340660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4209.004351] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4211.022539] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4212.391277] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4214.862060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4214.972487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4217.117511] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4219.062039] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4227.302370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4242.913669] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4242.914376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4242.918595] LustreError: Skipped 1 previous similar message [ 4242.924491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4242.928569] Lustre: Skipped 1 previous similar message [ 4243.951158] LustreError: 282664:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4243.992294] Lustre: server umount lustre-OST0000 complete [ 4248.033180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4248.038720] Lustre: Skipped 1 previous similar message [ 4248.041246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4248.043010] Lustre: Skipped 1 previous similar message [ 4251.586768] LustreError: 282865:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4251.589749] LustreError: 282865:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4251.626195] Lustre: server umount lustre-MDT0000 complete [ 4252.996525] LustreError: 279606:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764748834 with bad export cookie 9333266018705475895 [ 4252.998996] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4253.001653] LustreError: 279606:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4253.072308] LustreError: 283066:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4253.075038] LustreError: 283066:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4253.134971] Lustre: server umount lustre-MDT0001 complete [ 4256.839364] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 4257.940278] Key type lgssc unregistered [ 4258.079370] LNet: 283597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4258.081524] LNetError: 283597:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4258.089308] LNet: Removed LNI 192.168.201.137@tcp [ 4258.402331] Key type .llcrypt unregistered [ 4258.403584] Key type ._llcrypt unregistered [ 4266.070240] Key type ._llcrypt registered [ 4266.071831] Key type .llcrypt registered [ 4266.113938] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4266.470403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4266.522367] alg: No test for adler32 (adler32-zlib) [ 4267.394382] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 4267.493281] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 4269.087186] Key type lgssc registered [ 4269.501018] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4272.516722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4272.526090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4273.708933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4275.130870] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4278.239400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4278.356973] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4279.629939] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4281.513101] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4282.799272] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4285.093882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4285.203635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4287.173621] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4288.968934] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4298.534093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4303.841697] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4303.842043] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4303.845503] LustreError: Skipped 1 previous similar message [ 4303.851475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4303.855514] Lustre: Skipped 1 previous similar message [ 4305.199283] LustreError: 287234:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4305.236921] Lustre: server umount lustre-OST0000 complete [ 4308.610642] LustreError: 287436:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4308.612910] LustreError: 287436:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4308.639445] Lustre: server umount lustre-MDT0000 complete [ 4308.961423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4308.961467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4308.965015] Lustre: Skipped 1 previous similar message [ 4308.965703] LustreError: 284783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4308.977112] LustreError: 284783:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4309.966243] LustreError: 284756:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764748891 with bad export cookie 6314446940776836477 [ 4309.969460] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4309.971226] LustreError: 284756:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4310.043465] LustreError: 287636:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4310.046645] LustreError: 287636:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4310.104754] Lustre: server umount lustre-MDT0001 complete [ 4313.730045] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 4314.813916] Key type lgssc unregistered [ 4314.951438] LNet: 288167:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4314.953670] LNetError: 288167:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4314.961418] LNet: Removed LNI 192.168.201.137@tcp [ 4315.274329] Key type .llcrypt unregistered [ 4315.276108] Key type ._llcrypt unregistered [ 4319.159074] Key type ._llcrypt registered [ 4319.160613] Key type .llcrypt registered [ 4319.200478] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4325.285689] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 4331.388776] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4331.766943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4331.791491] alg: No test for adler32 (adler32-zlib) [ 4332.640843] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 4332.725731] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 4334.311205] Key type lgssc registered [ 4334.712485] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4338.000574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4340.394472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4342.672464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4345.008974] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4347.446021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4347.468191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4347.473157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4348.552778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4348.565765] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4348.602177] Lustre: lustre-MDT0000: new disk, initializing [ 4348.625175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4348.631513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4349.948085] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4354.464750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4354.493148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4354.535439] Lustre: 291652:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4354.563597] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4354.565930] Lustre: Skipped 1 previous similar message [ 4354.604311] Lustre: lustre-MDT0001: new disk, initializing [ 4354.641902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4354.656390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4354.660501] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4356.075147] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4359.240887] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4360.506488] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4362.973029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4363.000808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4363.112466] Lustre: lustre-OST0000: new disk, initializing [ 4363.114856] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4363.117933] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4363.160721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4365.297454] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4368.363787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4368.367719] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4368.393355] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4368.645809] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4370.225809] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4370.306762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4371.749761] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4371.826858] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4373.474198] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4373.474982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4373.475576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4373.479825] Lustre: Skipped 1 previous similar message [ 4373.483748] LustreError: Skipped 1 previous similar message [ 4374.575095] LustreError: 294595:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4374.612931] Lustre: server umount lustre-OST0000 complete [ 4378.592720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4378.593392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4378.598019] Lustre: Skipped 1 previous similar message [ 4378.600921] Lustre: Skipped 1 previous similar message [ 4383.712660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4383.715516] Lustre: Skipped 2 previous similar messages [ 4388.833120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4388.836360] Lustre: Skipped 1 previous similar message [ 4390.367097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4390.405604] LustreError: 294795:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 4390.407716] LustreError: 294795:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4390.439782] Lustre: server umount lustre-MDT0000 complete [ 4391.883505] LustreError: 293400:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764748973 with bad export cookie 13221785997630018959 [ 4391.887570] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4391.888327] LustreError: 293400:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4391.956879] LustreError: 294999:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 4391.959953] LustreError: 294999:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4392.028066] Lustre: server umount lustre-MDT0001 complete [ 4395.262878] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:02:55 (1764748975) [ 4397.852792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4398.058261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4399.632909] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4402.948781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4403.088815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4404.644908] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4406.644766] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4408.006599] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4410.504760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4412.765922] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4414.878728] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4428.256604] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4428.256607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4428.257935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4428.259970] LustreError: Skipped 1 previous similar message [ 4428.263216] Lustre: Skipped 1 previous similar message [ 4434.286093] LustreError: 297938:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4434.288726] LustreError: 297938:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4434.330366] Lustre: server umount lustre-OST0000 complete [ 4438.496762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4438.499991] Lustre: Skipped 1 previous similar message [ 4448.736404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4448.738587] Lustre: Skipped 7 previous similar messages [ 4450.271106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4450.309081] LustreError: 298139:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4450.311194] LustreError: 298139:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4450.337820] Lustre: server umount lustre-MDT0000 complete [ 4451.819254] LustreError: 298141:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764749033 with bad export cookie 13221785997630019827 [ 4451.821340] LustreError: 296391:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4451.822660] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4451.824267] LustreError: 298141:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4451.956102] Lustre: server umount lustre-MDT0001 complete [ 4455.906745] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 4457.019088] Key type lgssc unregistered [ 4457.160371] LNet: 298874:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4457.163991] LNetError: 298874:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4457.173383] LNet: Removed LNI 192.168.201.137@tcp [ 4457.499074] Key type .llcrypt unregistered [ 4457.500794] Key type ._llcrypt unregistered [ 4467.336487] Key type ._llcrypt registered [ 4467.338063] Key type .llcrypt registered [ 4467.379175] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 4473.863725] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4474.185528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4474.246271] alg: No test for adler32 (adler32-zlib) [ 4475.111087] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 4475.200225] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 4476.783144] Key type lgssc registered [ 4477.143360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4480.189472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4482.399520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4484.718223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4487.035937] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4488.790196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4490.502233] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4492.799509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4492.823127] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4492.827631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4493.908475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4493.921323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4493.956667] Lustre: lustre-MDT0000: new disk, initializing [ 4493.981358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4493.988746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4495.388994] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4499.702516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4499.727527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4499.758854] Lustre: 302652:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4499.783845] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4499.785635] Lustre: Skipped 1 previous similar message [ 4499.823680] Lustre: lustre-MDT0001: new disk, initializing [ 4499.861159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4499.880158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4499.884513] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4501.133218] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4504.285503] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4505.549934] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4507.964215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4507.989681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4508.092053] Lustre: lustre-OST2710: new disk, initializing [ 4508.094769] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4508.134513] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4510.076416] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4513.138064] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4514.808815] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4514.812504] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4514.851637] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4515.474140] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4515.504915] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4515.564908] Lustre: lustre-OST03e8: new disk, initializing [ 4515.567428] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4515.595772] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4517.455818] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4519.330156] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4520.448208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4520.452285] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4520.479065] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4521.936877] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4522.009588] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4523.319880] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4523.403450] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4524.822627] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4524.900912] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4526.317375] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4526.410881] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4530.658421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4530.659131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4530.661065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4530.664532] Lustre: Skipped 3 previous similar messages [ 4533.630827] LustreError: 306998:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4533.661601] Lustre: server umount lustre-MDT0000 complete [ 4534.988370] LustreError: 306911:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764749116 with bad export cookie 5187654311639904941 [ 4534.992625] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4534.993933] LustreError: 306911:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4535.066045] LustreError: 307200:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4535.068881] LustreError: 307200:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4535.122687] Lustre: server umount lustre-MDT0001 complete [ 4546.743198] LustreError: 307400:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4546.746506] LustreError: 307400:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4546.773266] Lustre: server umount lustre-OST2710 complete [ 4558.384743] LustreError: 307601:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4558.387046] LustreError: 307601:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4558.431771] Lustre: server umount lustre-OST03e8 complete [ 4564.472189] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 4567.744181] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4571.434630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4573.866287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4576.129386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4578.422225] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4580.853955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4580.885925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4580.973159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4580.985327] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4581.018441] Lustre: lustre-MDT0000: new disk, initializing [ 4581.041885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4581.048323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4582.505504] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4586.947079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4586.974978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4587.014458] Lustre: 310392:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4587.134499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4588.606850] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4591.876188] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4593.206057] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4595.697612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4595.719600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4595.817417] Lustre: lustre-OST0000: new disk, initializing [ 4595.819487] Lustre: Skipped 1 previous similar message [ 4595.821979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4595.824534] Lustre: Skipped 2 previous similar messages [ 4595.854626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4595.857275] Lustre: Skipped 1 previous similar message [ 4597.498914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4597.501935] Lustre: Skipped 1 previous similar message [ 4597.504463] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4597.546217] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4597.816365] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4601.038343] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4602.494110] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4602.576124] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4604.165419] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4604.267561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4607.972818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4607.973188] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4607.974085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4607.974090] Lustre: Skipped 3 previous similar messages [ 4607.976069] LustreError: Skipped 1 previous similar message [ 4607.982182] Lustre: Skipped 1 previous similar message [ 4611.124082] LustreError: 313332:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4611.126883] LustreError: 313332:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4611.174119] Lustre: server umount lustre-OST0000 complete [ 4613.089806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4613.089862] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4613.098836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4613.100584] Lustre: Skipped 2 previous similar messages [ 4618.208417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4618.211258] Lustre: Skipped 1 previous similar message [ 4618.710092] Lustre: server umount lustre-MDT0000 complete [ 4620.142475] LustreError: 310382:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764749201 with bad export cookie 5187654311639906607 [ 4620.146435] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4620.148016] LustreError: 310382:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4620.215774] LustreError: 313736:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4620.218677] LustreError: 313736:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4620.282677] Lustre: server umount lustre-MDT0001 complete [ 4623.515746] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:06:44 (1764749204) [ 4624.069536] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4624.639732] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:06:45 (1764749205) [ 4625.658543] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4628.412912] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4629.812674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4631.119960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4632.410827] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4632.782450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4635.092137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4635.193518] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4635.203473] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4635.268469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4636.577105] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4639.521286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4639.556206] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4639.567043] Lustre: 315864:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4641.068526] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4643.102305] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4644.512076] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4647.072481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4647.094738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4647.161932] Lustre: Denying initial registration attempt from nid 192.168.201.137@tcp, specified as failover [ 4647.165413] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4647.170900] LustreError: 317480:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 4647.175098] LustreError: 317480:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4647.178674] LustreError: 317480:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4647.203623] Lustre: server umount lustre-OST0000 complete [ 4647.205934] LustreError: 317480:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4649.951813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4649.954920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4649.958914] Lustre: Skipped 1 previous similar message [ 4649.961374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4655.683157] LustreError: 317848:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4655.685793] LustreError: 317848:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4655.719816] Lustre: server umount lustre-MDT0000 complete [ 4657.189619] LustreError: 315855:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764749238 with bad export cookie 5187654311639907475 [ 4657.192111] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4657.195207] LustreError: 315855:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4657.320671] Lustre: server umount lustre-MDT0001 complete [ 4661.437115] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 4662.574846] Key type lgssc unregistered [ 4662.707437] LNet: 318581:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4662.712072] LNetError: 318581:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4662.722358] LNet: Removed LNI 192.168.201.137@tcp [ 4663.046794] Key type .llcrypt unregistered [ 4663.048742] Key type ._llcrypt unregistered [ 4666.988083] Key type ._llcrypt registered [ 4666.989417] Key type .llcrypt registered [ 4667.032459] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4667.390724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4667.443567] alg: No test for adler32 (adler32-zlib) [ 4668.302239] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 4668.393415] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 4669.983554] Key type lgssc registered [ 4670.419389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4673.207281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4674.835707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4676.373190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4678.085560] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4678.579758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4686.467120] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4689.945949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4689.953575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4691.060447] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4691.073556] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4691.144079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4692.677301] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4695.737324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4695.766335] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4695.776586] Lustre: 321313:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4695.875671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4697.200938] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4699.095459] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4700.359313] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4703.081898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4703.113172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4703.194489] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4703.263644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4705.451591] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4707.626133] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4710.881178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4710.881920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4710.883630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4710.885674] LustreError: Skipped 1 previous similar message [ 4710.891062] Lustre: Skipped 1 previous similar message [ 4714.544252] LustreError: 323600:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4714.592283] Lustre: server umount lustre-OST0000 complete [ 4716.512370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4716.516242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4716.523040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4716.525255] Lustre: Skipped 1 previous similar message [ 4721.120144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4721.120618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4726.240441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4726.244419] Lustre: Skipped 2 previous similar messages [ 4730.335167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4730.374764] LustreError: 323801:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4730.376889] LustreError: 323801:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4730.405877] Lustre: server umount lustre-MDT0000 complete [ 4731.846291] LustreError: 322039:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764749313 with bad export cookie 9614236564094450224 [ 4731.849457] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4731.851701] LustreError: 322039:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4731.924502] LustreError: 324004:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4731.927416] LustreError: 324004:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4731.988237] Lustre: server umount lustre-MDT0001 complete [ 4736.168125] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 4737.378710] Key type lgssc unregistered [ 4737.513443] LNet: 324535:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4737.517063] LNetError: 324535:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4737.526371] LNet: Removed LNI 192.168.201.137@tcp [ 4737.886385] Key type .llcrypt unregistered [ 4737.888067] Key type ._llcrypt unregistered [ 4747.933341] Key type ._llcrypt registered [ 4747.934945] Key type .llcrypt registered [ 4747.982152] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4748.311581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4748.348023] alg: No test for adler32 (adler32-zlib) [ 4749.211149] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 4749.300697] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 4750.887251] Key type lgssc registered [ 4751.303148] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4754.295928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4754.304375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4755.519080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4757.051089] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4760.466523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4760.625619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4762.166041] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4764.281128] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4765.697340] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4768.284334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4768.430633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4770.706320] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4772.809912] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4787.169587] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4787.170522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4787.172878] Lustre: Skipped 2 previous similar messages [ 4787.175579] Lustre: Skipped 2 previous similar messages [ 4789.610173] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.37@tcp (stopping) [ 4792.289612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4792.292240] Lustre: Skipped 2 previous similar messages [ 4792.765839] LustreError: 328180:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4792.797263] Lustre: server umount lustre-MDT0000 complete [ 4794.110391] LustreError: 325888:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764749375 with bad export cookie 5817220126919706942 [ 4794.115174] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4794.115740] LustreError: 325888:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4794.722649] LustreError: 328070:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4794.729223] LustreError: 328070:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4794.729299] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [ 4797.407920] LustreError: 325903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4797.408224] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4799.455954] LustreError: 325903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4799.456589] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4799.464455] Lustre: Skipped 1 previous similar message [ 4800.257896] LustreError: 328383:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4800.260830] LustreError: 328383:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4800.324353] Lustre: server umount lustre-MDT0001 complete [ 4802.795342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4805.896424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4818.912661] LustreError: 325219:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001255cdb0 x1850473793478784/t0(0) o250->MGC192.168.201.137@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4819.029478] LustreError: 327518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4819.036166] LustreError: 327518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4819.056855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4819.086062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 4820.452082] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4823.585325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4823.730292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4824.739054] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4825.284119] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4827.179721] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4828.836741] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4831.813840] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 4833.030647] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4833.684771] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4839.394191] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4839.397996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4839.404658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4839.407343] Lustre: Skipped 2 previous similar messages [ 4841.138101] LustreError: 330684:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4841.140811] LustreError: 330684:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4841.182747] Lustre: server umount lustre-OST0000 complete [ 4844.512956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4844.513461] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4844.522437] Lustre: Skipped 2 previous similar messages [ 4848.764759] LustreError: 330886:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4848.766537] LustreError: 330886:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4848.795609] Lustre: server umount lustre-MDT0000 complete [ 4849.632252] LustreError: 330598:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4849.637378] LustreError: 330598:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4850.078218] LustreError: 325889:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764749431 with bad export cookie 5817220126919714096 [ 4850.080990] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4850.081932] LustreError: 325889:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4850.196134] Lustre: server umount lustre-MDT0001 complete [ 4853.866941] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 4854.909689] Key type lgssc unregistered [ 4855.034823] LNet: 331617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4855.037524] LNetError: 331617:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4855.045428] LNet: Removed LNI 192.168.201.137@tcp [ 4855.366670] Key type .llcrypt unregistered [ 4855.368416] Key type ._llcrypt unregistered [ 4864.899895] Key type ._llcrypt registered [ 4864.901370] Key type .llcrypt registered [ 4864.938908] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4865.280465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4865.344976] alg: No test for adler32 (adler32-zlib) [ 4866.228973] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 4866.311838] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 4867.895188] Key type lgssc registered [ 4868.287770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4871.148747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4871.156690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4872.318152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4873.689892] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4876.585605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4876.704734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4878.104287] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4880.008233] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4881.341152] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4886.561969] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4886.566801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4886.569158] Lustre: Skipped 1 previous similar message [ 4888.571345] LustreError: 334383:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4888.597046] Lustre: server umount lustre-MDT0000 complete [ 4889.792078] LustreError: 332973:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764749471 with bad export cookie 3282591878622241491 [ 4889.795905] LustreError: 332973:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4889.796711] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4889.857941] LustreError: 334586:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4889.860839] LustreError: 334586:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4889.909877] Lustre: server umount lustre-MDT0001 complete [ 4894.694856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4894.794469] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4894.806727] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4894.881187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4896.215358] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4899.134178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4899.165754] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4899.176992] Lustre: 335356:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4899.268091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4900.618932] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4902.456172] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4903.664564] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4906.688159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4906.781221] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4906.785730] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4906.788881] LustreError: 337012:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 4906.791848] LustreError: 337012:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4906.794032] LustreError: 337012:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4906.819600] Lustre: server umount lustre-OST0000 complete [ 4906.821416] LustreError: 337012:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4909.994876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4910.069764] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4910.142279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4911.900920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 4912.340892] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4914.408413] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4917.512867] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4917.542342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4917.571641] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4917.605572] Lustre: lustre-OST0001: new disk, initializing [ 4917.608613] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4919.652064] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 4919.877158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4919.880583] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4919.913154] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4921.547060] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4924.901948] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4924.906431] Lustre: Skipped 1 previous similar message [ 4924.908242] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4924.908422] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4924.913436] Lustre: Skipped 1 previous similar message [ 4928.686332] LustreError: 339135:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4928.692277] LustreError: 339135:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4928.722502] Lustre: server umount lustre-OST0001 complete [ 4930.017225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4930.018079] LustreError: 337769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4930.021016] Lustre: Skipped 1 previous similar message [ 4930.025456] LustreError: 337769:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4930.028074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4930.032555] Lustre: Skipped 1 previous similar message [ 4935.137677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4935.138557] LustreError: 339337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4935.139726] Lustre: Skipped 1 previous similar message [ 4935.143636] LustreError: 339337:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4940.258315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4940.258683] LustreError: 337479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4940.260693] Lustre: Skipped 1 previous similar message [ 4940.265911] LustreError: 337479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4944.351113] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4944.368081] LustreError: 339336:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4944.370194] LustreError: 339336:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4944.402320] Lustre: server umount lustre-OST0000 complete [ 4950.496744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4950.496797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4950.497508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4950.508045] Lustre: Skipped 2 previous similar messages [ 4960.223113] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4960.256226] LustreError: 339539:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4960.258488] LustreError: 339539:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4960.286306] Lustre: server umount lustre-MDT0000 complete [ 4960.735904] LustreError: 336085:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4960.741385] LustreError: 336085:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4961.514524] LustreError: 335348:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764749543 with bad export cookie 3282591878622242030 [ 4961.519118] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4961.645516] Lustre: server umount lustre-MDT0001 complete [ 4965.211494] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 4966.304554] Key type lgssc unregistered [ 4966.433383] LNet: 340274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4966.436275] LNetError: 340274:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4966.443287] LNet: Removed LNI 192.168.201.137@tcp [ 4966.728356] Key type .llcrypt unregistered [ 4966.729597] Key type ._llcrypt unregistered [ 4977.627814] Key type ._llcrypt registered [ 4977.628887] Key type .llcrypt registered [ 4977.663197] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 4984.258233] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 4984.651286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4984.678827] alg: No test for adler32 (adler32-zlib) [ 4985.532844] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 4985.614859] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 4987.199156] Key type lgssc registered [ 4987.580598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4990.713753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4993.128132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4995.418865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4997.835996] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5000.230039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5000.246223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5000.252565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5001.345951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5001.362616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5001.399914] Lustre: lustre-MDT0000: new disk, initializing [ 5001.428555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5001.439045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5002.824806] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5007.167181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5007.190818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5007.227887] Lustre: 344198:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5007.251889] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5007.254095] Lustre: Skipped 1 previous similar message [ 5007.288304] Lustre: lustre-MDT0001: new disk, initializing [ 5007.326118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5007.340100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5007.343560] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5008.666535] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5011.773304] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5012.980209] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5015.194589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5015.220217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5015.309886] Lustre: lustre-OST0000: new disk, initializing [ 5015.311881] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5015.343820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5016.826785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5016.830350] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5016.862873] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5017.210411] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5020.375920] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5021.839970] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5021.919530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5023.352173] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5023.434504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5027.297829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5027.298369] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5027.302097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5027.305061] Lustre: Skipped 1 previous similar message [ 5027.308135] Lustre: Skipped 1 previous similar message [ 5030.190102] LustreError: 347135:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5030.230353] Lustre: server umount lustre-OST0000 complete [ 5032.417730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5032.418808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5032.423059] Lustre: Skipped 1 previous similar message [ 5032.427764] Lustre: Skipped 1 previous similar message [ 5037.536445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5037.538586] Lustre: Skipped 1 previous similar message [ 5037.693362] LustreError: 347336:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5037.695348] LustreError: 347336:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5037.724399] Lustre: server umount lustre-MDT0000 complete [ 5038.964235] LustreError: 344190:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764749620 with bad export cookie 16163642107908123260 [ 5038.967200] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5038.968901] LustreError: 344190:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5039.028995] LustreError: 347537:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5039.031111] LustreError: 347537:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5039.082957] Lustre: server umount lustre-MDT0001 complete [ 5041.922256] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 03:13:42 (1764749622) [ 5043.682816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5045.493553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5052.445176] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 5055.455656] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5058.493310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5060.630530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5062.495554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5064.561148] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5066.659275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5066.679987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5066.760249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5066.772206] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5066.812870] Lustre: lustre-MDT0000: new disk, initializing [ 5066.835965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5066.842734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5068.041615] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5072.352129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5072.376925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5072.413778] Lustre: 351302:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5072.441957] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5072.444082] Lustre: Skipped 1 previous similar message [ 5072.485639] Lustre: lustre-MDT0001: new disk, initializing [ 5072.527748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5072.541805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5072.547554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5073.921639] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5077.170266] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5078.485364] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5080.777395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5080.802191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5080.903136] Lustre: lustre-OST0000: new disk, initializing [ 5080.905781] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5080.937801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5082.681299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5082.684344] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5082.721981] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5082.930157] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5086.229963] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5087.676712] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5087.760759] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5089.191267] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5089.270275] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5092.014195] LustreError: 354238:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5092.016547] LustreError: 354238:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5092.057073] Lustre: server umount lustre-OST0000 complete [ 5097.952953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5097.953763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5097.959064] Lustre: Skipped 1 previous similar message [ 5103.072736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5103.075298] Lustre: Skipped 2 previous similar messages [ 5103.075962] LustreError: 354439:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5103.078782] LustreError: 354439:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5103.103498] Lustre: server umount lustre-MDT0000 complete [ 5104.373941] LustreError: 351293:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764749685 with bad export cookie 16163642107908124128 [ 5104.377524] LustreError: 351293:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5104.377696] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5113.828304] LustreError: 354642:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5113.830250] LustreError: 354642:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5113.885479] Lustre: server umount lustre-MDT0001 complete [ 5116.779848] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 03:14:57 (1764749697) [ 5119.278494] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 03:15:00 (1764749700) [ 5122.176661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5122.363836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5123.791582] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5126.793822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5128.422863] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5130.381045] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5131.756464] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5134.013278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5135.974060] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5137.790595] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5151.991970] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5152.600735] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5153.187833] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5153.878489] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5154.569127] Lustre: Failing over lustre-MDT0000 [ 5154.770065] LustreError: 358119:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5154.772180] LustreError: 358119:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5154.803980] Lustre: server umount lustre-MDT0000 complete [ 5157.858326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5157.858511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5157.859144] LustreError: 357273:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5157.859150] LustreError: 357273:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5157.871638] Lustre: Skipped 1 previous similar message [ 5158.761655] LustreError: 355653:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.37@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5158.766851] LustreError: 355653:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5162.977473] LustreError: 357762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5162.983776] LustreError: 357762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5167.902474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5167.945421] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5168.057835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5168.060326] Lustre: Skipped 2 previous similar messages [ 5168.083787] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5168.994903] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5169.536846] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5173.229469] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5173.235572] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5173.254303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5174.512636] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5175.126357] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5177.034525] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5177.741934] Lustre: Failing over lustre-MDT0000 [ 5177.937662] Lustre: server umount lustre-MDT0000 complete [ 5178.338923] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5178.339549] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5178.342774] LustreError: 357768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5178.346844] Lustre: Skipped 3 previous similar messages [ 5178.353408] LustreError: 357768:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5183.456879] LustreError: 357768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5183.462784] LustreError: 357768:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5191.171559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5191.222737] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5191.354781] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5192.839918] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5194.595061] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5196.784136] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5196.786406] Lustre: Skipped 2 previous similar messages [ 5196.795255] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5196.818583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5197.971465] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5198.541423] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5201.890864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5201.892309] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5201.892985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5201.898212] Lustre: Skipped 2 previous similar messages [ 5206.990657] LustreError: 360225:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5206.995143] LustreError: 360225:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5207.009378] LustreError: 357768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5207.015717] LustreError: 357768:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5207.027468] Lustre: server umount lustre-MDT0000 complete [ 5208.324287] LustreError: 355639:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764749789 with bad export cookie 16163642107908126767 [ 5208.327744] LustreError: 355639:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5208.328155] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5208.446147] Lustre: server umount lustre-MDT0001 complete [ 5220.065221] Lustre: server umount lustre-OST0000 complete [ 5225.278725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5225.428977] LustreError: 361454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5226.771487] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5229.726407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5231.219158] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5233.054133] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5234.361325] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5236.804528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5236.908588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5236.911694] Lustre: Skipped 3 previous similar messages [ 5238.808508] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5240.629286] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5250.214081] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5255.653734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5255.655232] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5255.678047] Lustre: Skipped 1 previous similar message [ 5255.683269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5255.686468] Lustre: Skipped 3 previous similar messages [ 5258.932589] Lustre: server umount lustre-OST0000 complete [ 5265.889651] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5265.893537] LustreError: Skipped 1 previous similar message [ 5267.289653] LustreError: 363925:0:(ldlm_resource.c:981:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000c8f7ee83) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5267.356748] Lustre: server umount lustre-MDT0000 complete [ 5269.603926] LustreError: 361435:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764749851 with bad export cookie 16163642107908127495 [ 5269.608170] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5269.618939] LustreError: 361435:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5269.877341] Lustre: server umount lustre-MDT0001 complete [ 5274.915442] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 5276.033955] Key type lgssc unregistered [ 5276.176427] LNet: 364658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5276.179033] LNetError: 364658:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5276.185345] LNet: Removed LNI 192.168.201.137@tcp [ 5276.496722] Key type .llcrypt unregistered [ 5276.498541] Key type ._llcrypt unregistered [ 5286.219160] Key type ._llcrypt registered [ 5286.220625] Key type .llcrypt registered [ 5286.267868] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5286.600667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5286.613197] alg: No test for adler32 (adler32-zlib) [ 5287.483128] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 5287.578526] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 5289.159100] Key type lgssc registered [ 5289.583746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5292.286946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5292.293704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5293.454209] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5294.824787] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5297.813579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5297.937021] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5299.152664] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5300.847221] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5301.991669] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5304.118482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5304.220318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5305.940826] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5307.792790] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5317.543247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5323.744414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5323.747223] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5323.751562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5327.840877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5327.841576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5327.845818] Lustre: Skipped 1 previous similar message [ 5329.469301] LustreError: 368399:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5329.494748] Lustre: server umount lustre-MDT0000 complete [ 5330.818229] LustreError: 366158:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764749912 with bad export cookie 9620781067876222494 [ 5330.821786] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5330.823052] LustreError: 366158:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5330.883750] LustreError: 368601:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5330.885566] LustreError: 368601:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5330.943627] Lustre: server umount lustre-MDT0001 complete [ 5342.448757] LustreError: 368801:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5342.451088] LustreError: 368801:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5342.495504] Lustre: server umount lustre-OST0000 complete [ 5347.153101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5347.291768] LustreError: 369531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5347.300052] LustreError: 369531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5347.315029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5348.551406] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5351.198488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5352.559102] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5354.140869] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5355.313937] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5357.575686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5357.693398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5357.695880] Lustre: Skipped 1 previous similar message [ 5359.774456] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5361.570633] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5373.093714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5387.231832] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5387.234901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5387.239519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5387.241224] Lustre: Skipped 2 previous similar messages [ 5390.206666] LustreError: 371897:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5390.208496] LustreError: 371897:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5390.239569] Lustre: server umount lustre-MDT0000 complete [ 5391.425197] LustreError: 371900:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764749972 with bad export cookie 9620781067876223600 [ 5391.427896] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5391.428917] LustreError: 371900:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5391.529148] Lustre: server umount lustre-MDT0001 complete [ 5403.055471] LustreError: 372302:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5403.057329] LustreError: 372302:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5403.097299] Lustre: server umount lustre-OST0000 complete [ 5408.275237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5408.439369] LustreError: 373128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5408.466870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5409.853714] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5412.981866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5414.449565] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5416.296852] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5417.515898] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5419.861944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5421.938841] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5424.099316] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5433.325279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5438.436131] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5438.437207] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5438.438065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5438.438072] Lustre: Skipped 3 previous similar messages [ 5438.440521] Lustre: Skipped 3 previous similar messages [ 5438.443404] LustreError: Skipped 1 previous similar message [ 5442.162707] LustreError: 375391:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5442.165685] LustreError: 375391:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5442.224287] Lustre: server umount lustre-OST0000 complete [ 5444.064364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5444.068052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5444.074082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5444.075860] Lustre: Skipped 1 previous similar message [ 5453.791849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5453.793935] Lustre: Skipped 3 previous similar messages [ 5458.399139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5458.440886] LustreError: 375593:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5458.443689] LustreError: 375593:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5458.475940] Lustre: server umount lustre-MDT0000 complete [ 5458.913581] LustreError: 373128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5458.921175] LustreError: 373128:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5459.914489] LustreError: 373109:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764750041 with bad export cookie 9620781067876224608 [ 5459.916964] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5459.919920] LustreError: 373109:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5460.060958] Lustre: server umount lustre-MDT0001 complete [ 5464.188992] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 5465.369451] Key type lgssc unregistered [ 5465.508838] LNet: 376327:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5465.512690] LNetError: 376327:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5465.526472] LNet: Removed LNI 192.168.201.137@tcp [ 5465.885074] Key type .llcrypt unregistered [ 5465.886882] Key type ._llcrypt unregistered [ 5476.319654] Key type ._llcrypt registered [ 5476.321260] Key type .llcrypt registered [ 5476.365743] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5476.802750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5476.866268] alg: No test for adler32 (adler32-zlib) [ 5477.736653] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 5477.829849] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 5479.415179] Key type lgssc registered [ 5479.884108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5482.986731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5482.992218] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5482.996122] LustreError: 377725:0:(osd_handler.c:8561:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5483.006309] LustreError: 377725:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5483.010030] LustreError: 377725:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5483.013542] LustreError: 377725:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5483.017586] LustreError: 377725:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5485.869404] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5485.873242] LustreError: 378122:0:(osd_handler.c:8561:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5485.886644] LustreError: 378122:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5485.889544] LustreError: 378122:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5485.892783] LustreError: 378122:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5485.897719] LustreError: 378122:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5493.106457] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 5494.245390] Key type lgssc unregistered [ 5494.377468] LNet: 379107:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5494.380915] LNetError: 379107:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5494.389327] LNet: Removed LNI 192.168.201.137@tcp [ 5494.684588] Key type .llcrypt unregistered [ 5494.686155] Key type ._llcrypt unregistered [ 5500.901792] Key type ._llcrypt registered [ 5500.903346] Key type .llcrypt registered [ 5500.941141] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 5507.506703] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5507.891984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5507.910113] alg: No test for adler32 (adler32-zlib) [ 5508.773971] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 5508.866813] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 5510.455168] Key type lgssc registered [ 5510.881925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5514.345630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5517.087670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5519.634296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5522.111469] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5524.692367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5524.714831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5524.722079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5525.823548] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5525.842445] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5525.881422] Lustre: lustre-MDT0000: new disk, initializing [ 5525.911067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5525.919800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5527.446394] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5531.983565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5532.004556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5532.038171] Lustre: 382210:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5532.061512] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5532.063201] Lustre: Skipped 1 previous similar message [ 5532.095588] Lustre: lustre-MDT0001: new disk, initializing [ 5532.131396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5532.143072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5532.146632] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5533.563229] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5537.107659] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5538.468622] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5541.030364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5541.059808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5541.174655] Lustre: lustre-OST0000: new disk, initializing [ 5541.177305] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5541.215185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5543.268470] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5544.701580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5544.705964] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5544.735109] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5546.551212] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5548.079402] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5548.162959] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5549.660452] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5549.739894] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5552.493870] LustreError: 385146:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5552.536094] Lustre: server umount lustre-OST0000 complete [ 5555.168420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5555.169456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5555.172301] Lustre: Skipped 1 previous similar message [ 5560.288278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5560.291020] Lustre: Skipped 2 previous similar messages [ 5564.389623] LustreError: 385347:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5564.392154] LustreError: 385347:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5564.428599] Lustre: server umount lustre-MDT0000 complete [ 5565.911424] LustreError: 382991:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764750147 with bad export cookie 15397420661680160556 [ 5565.914080] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5565.918841] LustreError: 382991:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5576.163761] LustreError: 385551:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5576.166199] LustreError: 385551:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5576.218576] Lustre: server umount lustre-MDT0001 complete [ 5579.531187] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5580.247578] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 03:22:40 (1764750160) [ 5582.745597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5582.915754] LustreError: 386275:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5582.923210] LustreError: 386275:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5582.941700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5584.378794] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5587.556598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5587.690759] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5589.184666] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5590.972434] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5592.312893] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5594.919922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5597.315584] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5599.324440] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5601.739801] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5601.761422] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5601.816765] Lustre: lustre-OST0001: new disk, initializing [ 5601.818707] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5601.849675] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5601.852829] Lustre: Skipped 1 previous similar message [ 5602.206105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5602.212175] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5602.245342] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5603.795646] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5607.114719] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5612.512036] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5612.512348] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5612.515827] LustreError: Skipped 1 previous similar message [ 5612.516485] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5612.521136] Lustre: Skipped 1 previous similar message [ 5612.527214] Lustre: Skipped 1 previous similar message [ 5615.218134] LustreError: 389473:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5615.221264] LustreError: 389473:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5615.244483] Lustre: server umount lustre-OST0001 complete [ 5618.661041] LustreError: 389475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5618.668205] LustreError: 389475:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5623.780493] LustreError: 387868:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5623.787591] LustreError: 387868:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5628.898916] LustreError: 387868:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5628.899606] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5628.900732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5628.902630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5628.902635] Lustre: Skipped 3 previous similar messages [ 5628.906139] LustreError: 387868:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5628.909845] LustreError: Skipped 1 previous similar message [ 5633.711543] LustreError: 389697:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5633.714541] LustreError: 389697:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5633.758519] Lustre: server umount lustre-OST0000 complete [ 5639.136921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5639.136984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5639.141587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5639.147198] Lustre: Skipped 2 previous similar messages [ 5639.152567] Lustre: Skipped 2 previous similar messages [ 5641.403590] Lustre: server umount lustre-MDT0000 complete [ 5642.982124] LustreError: 386256:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764750224 with bad export cookie 15397420661680161424 [ 5642.983975] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5642.990032] LustreError: 386256:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5649.153487] LustreError: 390099:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5649.155941] LustreError: 390099:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5649.218418] Lustre: server umount lustre-MDT0001 complete [ 5653.477034] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 5654.571348] Key type lgssc unregistered [ 5654.707505] LNet: 390631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5654.711213] LNetError: 390631:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5654.720350] LNet: Removed LNI 192.168.201.137@tcp [ 5655.043548] Key type .llcrypt unregistered [ 5655.045218] Key type ._llcrypt unregistered [ 5667.062648] Key type ._llcrypt registered [ 5667.064183] Key type .llcrypt registered [ 5667.103350] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 5673.798848] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5674.277781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5674.293102] alg: No test for adler32 (adler32-zlib) [ 5675.157858] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 5675.248797] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 5676.839174] Key type lgssc registered [ 5677.232832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5680.373640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5682.717544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5684.949427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5687.321547] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5689.857992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5689.880869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5689.885677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5690.982184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5690.994418] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5691.036168] Lustre: lustre-MDT0000: new disk, initializing [ 5691.063819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5691.072504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5692.558816] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5697.114082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5697.140169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5697.181739] Lustre: 394567:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5697.208247] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5697.210814] Lustre: Skipped 1 previous similar message [ 5697.247741] Lustre: lustre-MDT0001: new disk, initializing [ 5697.285280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5697.301650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5697.307075] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5698.743359] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5702.129194] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5703.495028] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5706.013928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5706.042386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5706.138102] Lustre: lustre-OST0000: new disk, initializing [ 5706.140677] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5706.175592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5706.754862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5706.761083] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5706.811441] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5708.265667] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5711.530835] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5713.004591] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5713.085594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5714.579258] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5714.665279] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5716.961388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5716.962465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5716.964454] LustreError: Skipped 1 previous similar message [ 5716.970605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5716.975160] Lustre: Skipped 1 previous similar message [ 5717.421511] LustreError: 397500:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 5717.456958] Lustre: server umount lustre-OST0000 complete [ 5722.080640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5722.081506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5722.084796] Lustre: Skipped 2 previous similar messages [ 5722.087711] Lustre: Skipped 1 previous similar message [ 5727.200247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5727.202395] Lustre: Skipped 1 previous similar message [ 5732.320233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5732.323148] Lustre: Skipped 1 previous similar message [ 5733.343210] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5733.385492] LustreError: 397702:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5733.388926] LustreError: 397702:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5733.425045] Lustre: server umount lustre-MDT0000 complete [ 5734.767971] LustreError: 397703:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764750316 with bad export cookie 15795337611780844981 [ 5734.771127] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5734.774685] LustreError: 397703:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5734.838444] LustreError: 397905:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 5734.841676] LustreError: 397905:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5734.899168] Lustre: server umount lustre-MDT0001 complete [ 5738.419469] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 03:25:19 (1764750319) [ 5741.958779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5745.144529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5745.312110] Lustre: lustre-MDT0000: new disk, initializing [ 5745.351730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5746.825310] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5749.821113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5749.941441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5751.293840] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5753.230230] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5754.661618] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5759.520811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5759.521512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5759.526603] Lustre: Skipped 1 previous similar message [ 5761.530522] LustreError: 400513:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5761.532352] LustreError: 400513:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5761.558079] Lustre: server umount lustre-MDT0000 complete [ 5762.750571] LustreError: 399151:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764750344 with bad export cookie 15795337611780845786 [ 5762.753413] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5762.753473] LustreError: 399151:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5762.866102] Lustre: server umount lustre-MDT0001 complete [ 5766.142109] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 03:25:46 (1764750346) [ 5768.449422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5768.608325] LustreError: 401435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5768.615938] LustreError: 401435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5768.633631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5770.017640] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5772.969643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5774.486421] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5776.426809] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5777.857870] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5780.301928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5782.333430] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5784.017353] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5797.513631] Lustre: Permanently deactivating lustre-OST0000 [ 5797.520112] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5806.064509] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5807.342529] LustreError: 404244:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5807.348218] LustreError: 404244:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 5807.351739] LustreError: 404244:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5808.100210] LustreError: 404292:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 5808.103601] LustreError: 404292:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5814.957146] LustreError: 404389:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5814.960062] LustreError: 404389:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5814.999821] Lustre: server umount lustre-OST0000 complete [ 5815.924811] LustreError: 404541:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 5815.928425] LustreError: 404541:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5819.361356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5819.361876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5819.362378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5819.362383] Lustre: Skipped 1 previous similar message [ 5819.365618] Lustre: Skipped 1 previous similar message [ 5826.530866] LustreError: 404638:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5826.533544] LustreError: 404638:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5826.560310] Lustre: server umount lustre-MDT0000 complete [ 5827.818155] LustreError: 401461:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764750409 with bad export cookie 15795337611780846332 [ 5827.821574] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5827.823646] LustreError: 401461:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5838.347369] Lustre: server umount lustre-MDT0001 complete [ 5841.323703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5842.653935] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5843.569945] LustreError: 405682:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5845.763906] Lustre: 406020:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.137@tcp [ 5846.504363] LustreError: 406118:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5846.507020] LustreError: 406118:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5846.542723] Lustre: server umount MGS complete [ 5850.531447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5850.689473] LustreError: 406746:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5850.696181] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5850.700955] Lustre: Skipped 1 previous similar message [ 5850.722168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5850.725358] Lustre: Skipped 2 previous similar messages [ 5852.134558] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5855.049782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5855.158201] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5856.571592] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5858.506709] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5859.903321] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5860.598202] Lustre: Permanently reactivating lustre-OST0000 [ 5860.603393] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5860.606434] Lustre: Skipped 2 previous similar messages [ 5884.383064] LustreError: 408254:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5886.882344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5887.009355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5887.012169] Lustre: Skipped 1 previous similar message [ 5889.041355] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5891.051698] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5904.354861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5904.355591] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5904.355807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5904.355812] Lustre: Skipped 3 previous similar messages [ 5904.360582] Lustre: Skipped 1 previous similar message [ 5905.325063] LustreError: 410034:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5905.327237] LustreError: 410034:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5905.367951] Lustre: server umount lustre-OST0000 complete [ 5908.891090] Lustre: server umount lustre-MDT0000 complete [ 5909.473308] LustreError: 407458:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5909.478636] LustreError: 407458:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5910.220317] LustreError: 406726:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764750491 with bad export cookie 15795337611780847277 [ 5910.222201] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5910.223663] LustreError: 406726:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5910.341189] Lustre: server umount lustre-MDT0001 complete [ 5914.051637] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 5915.146705] Key type lgssc unregistered [ 5915.286760] LNet: 410973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5915.290724] LNetError: 410973:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5915.302599] LNet: Removed LNI 192.168.201.137@tcp [ 5915.645210] Key type .llcrypt unregistered [ 5915.646472] Key type ._llcrypt unregistered [ 5922.534509] Key type ._llcrypt registered [ 5922.536114] Key type .llcrypt registered [ 5922.579990] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 5929.160043] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 5929.586689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5929.600547] alg: No test for adler32 (adler32-zlib) [ 5930.456308] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 5930.553773] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 5932.135180] Key type lgssc registered [ 5932.537372] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5935.663193] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5937.872764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5940.014753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5942.235633] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5944.655025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5944.675029] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5944.681991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5945.775064] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5945.785806] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5945.819491] Lustre: lustre-MDT0000: new disk, initializing [ 5945.846076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5945.852210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5947.191724] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5951.381128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5951.405372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5951.437678] Lustre: 414078:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5951.474194] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5951.476987] Lustre: Skipped 1 previous similar message [ 5951.513313] Lustre: lustre-MDT0001: new disk, initializing [ 5951.546069] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5951.557855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5951.561854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5952.764995] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5955.865986] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5957.146178] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5959.548337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5959.578131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5959.676531] Lustre: lustre-OST0000: new disk, initializing [ 5959.679712] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5959.714815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5961.714408] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 5964.963497] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5966.372147] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5966.842194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5966.845731] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5966.879919] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5967.467701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5968.840299] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5968.924854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5971.939698] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5971.939890] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5971.945054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5971.947672] LustreError: Skipped 1 previous similar message [ 5975.662138] LustreError: 417026:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5975.707365] Lustre: server umount lustre-OST0000 complete [ 5977.056844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5977.056973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5977.057777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5977.057781] Lustre: Skipped 1 previous similar message [ 5977.069581] Lustre: Skipped 2 previous similar messages [ 5982.176276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5982.178782] Lustre: Skipped 2 previous similar messages [ 5987.296083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5987.298206] Lustre: Skipped 1 previous similar message [ 5991.391156] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5991.427409] LustreError: 417227:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5991.430367] LustreError: 417227:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5991.456946] Lustre: server umount lustre-MDT0000 complete [ 5992.416541] LustreError: 414088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5992.424615] LustreError: 414088:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5992.831135] LustreError: 414857:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764750574 with bad export cookie 1679589454707841833 [ 5992.834311] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5992.836903] LustreError: 414857:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5992.911224] LustreError: 417429:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5992.914093] LustreError: 417429:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5992.971766] Lustre: server umount lustre-MDT0001 complete [ 5995.889847] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5996.450348] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 03:29:37 (1764750577) [ 5998.829326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5998.993197] LustreError: 418148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5999.019585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6000.400189] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6003.314371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6003.436387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6004.819908] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6006.758097] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6008.008961] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6010.337139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6012.521143] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6014.399168] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6015.057789] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6026.155992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6028.255792] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6028.259252] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6028.263921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6033.710094] LustreError: 420605:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6033.713239] LustreError: 420605:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6033.752410] Lustre: server umount lustre-OST0000 complete [ 6039.520541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6039.520644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6039.521282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6039.521286] Lustre: Skipped 2 previous similar messages [ 6039.524841] Lustre: Skipped 2 previous similar messages [ 6039.527504] LustreError: Skipped 1 previous similar message [ 6049.247123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6049.276274] LustreError: 420807:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6049.278382] LustreError: 420807:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6049.299803] Lustre: server umount lustre-MDT0000 complete [ 6049.759826] LustreError: 418148:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6049.763643] LustreError: 418148:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6050.507938] LustreError: 419763:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764750632 with bad export cookie 1679589454707842715 [ 6050.510671] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6050.511325] LustreError: 419763:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6050.620411] Lustre: server umount lustre-MDT0001 complete [ 6054.068305] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6055.015503] Key type lgssc unregistered [ 6055.147340] LNet: 421540:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6055.149693] LNetError: 421540:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6055.161290] LNet: Removed LNI 192.168.201.137@tcp [ 6055.443455] Key type .llcrypt unregistered [ 6055.444511] Key type ._llcrypt unregistered [ 6062.624321] Key type ._llcrypt registered [ 6062.625368] Key type .llcrypt registered [ 6062.661484] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6062.758271] Key type .llcrypt unregistered [ 6062.759586] Key type ._llcrypt unregistered [ 6069.593107] Key type ._llcrypt registered [ 6069.594306] Key type .llcrypt registered [ 6069.630889] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6069.961310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6070.026829] alg: No test for adler32 (adler32-zlib) [ 6070.872449] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 6070.950593] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6072.535130] Key type lgssc registered [ 6072.871792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6075.350528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6075.357968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6076.504034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6077.757366] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6080.491679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6080.601522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6082.407447] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6084.116395] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6085.600598] LustreError: 423703:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6085.608219] LustreError: 423703:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6086.373341] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6086.491643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6087.527952] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6087.834979] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6091.744851] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6091.748289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6091.755299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6092.772721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6092.773816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6092.780476] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6092.912109] LustreError: 425755:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6092.953436] Lustre: server umount lustre-OST0000 complete [ 6095.841386] LustreError: 423682:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764750677 with bad export cookie 15997069577019060022 [ 6095.842206] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6095.847175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6095.853214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6095.854881] Lustre: Skipped 1 previous similar message [ 6096.511742] LustreError: 425957:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6096.514625] LustreError: 425957:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6096.546599] Lustre: server umount lustre-MDT0000 complete [ 6097.865206] LustreError: 423682:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764750679 with bad export cookie 15997069577019059532 [ 6097.869736] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6097.929045] LustreError: 426159:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6097.932088] LustreError: 426159:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6097.986375] Lustre: server umount lustre-MDT0001 complete [ 6101.758193] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6102.816792] Key type lgssc unregistered [ 6102.950426] LNet: 426689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6102.952939] LNetError: 426689:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6102.958340] LNet: Removed LNI 192.168.201.137@tcp [ 6103.280532] Key type .llcrypt unregistered [ 6103.281885] Key type ._llcrypt unregistered [ 6112.687297] Key type ._llcrypt registered [ 6112.688637] Key type .llcrypt registered [ 6112.727341] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6113.054081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6113.060778] alg: No test for adler32 (adler32-zlib) [ 6113.917706] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 6114.007154] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6115.593733] Key type lgssc registered [ 6115.986655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6118.726356] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6118.733606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6134.175458] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6139.295214] LustreError: 428036:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.137@tcp: failed processing log, type 4: rc = -110 [ 6165.919224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6167.976743] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6169.857394] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6172.328035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6172.505238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6172.535296] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6174.025376] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6177.189199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6177.307027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6178.636882] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6180.447846] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6181.686868] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6186.465749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6186.467314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6186.471529] Lustre: Skipped 1 previous similar message [ 6191.085105] LustreError: 430315:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 6191.124544] Lustre: server umount lustre-OST0000 complete [ 6192.608153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6192.611678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6192.617876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6192.620488] Lustre: Skipped 1 previous similar message [ 6196.704091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6196.704603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6198.596641] LustreError: 430516:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6198.599720] LustreError: 430516:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6198.631747] Lustre: server umount lustre-MDT0000 complete [ 6199.992262] LustreError: 430228:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764750781 with bad export cookie 16140999633379104979 [ 6199.995610] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6199.996911] LustreError: 430228:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6200.066684] LustreError: 430719:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6200.068976] LustreError: 430719:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6200.127959] Lustre: server umount lustre-MDT0001 complete [ 6203.921781] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6205.034075] Key type lgssc unregistered [ 6205.175234] LNet: 431250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6205.178448] LNetError: 431250:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6205.188500] LNet: Removed LNI 192.168.201.137@tcp [ 6205.520524] Key type .llcrypt unregistered [ 6205.522221] Key type ._llcrypt unregistered [ 6215.116769] Key type ._llcrypt registered [ 6215.118371] Key type .llcrypt registered [ 6215.158923] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6215.584345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6215.612914] alg: No test for adler32 (adler32-zlib) [ 6216.465283] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 6216.542861] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6218.119175] Key type lgssc registered [ 6218.529189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6221.281935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6221.290184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6222.479476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6223.817115] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6226.815042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6226.956820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6228.366803] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6230.268732] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6231.522366] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6233.882166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6233.996699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6235.966589] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6237.879445] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6248.358788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6252.512207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6252.514458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6252.518049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6253.537865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6253.538192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6253.542117] Lustre: Skipped 1 previous similar message [ 6253.544958] Lustre: Skipped 1 previous similar message [ 6256.383305] LustreError: 434894:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6256.412827] Lustre: server umount lustre-MDT0000 complete [ 6257.215126] LustreError: 432604:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.37@tcp arrived at 1764750838 with bad export cookie 7815164722730063996 [ 6257.221852] LustreError: 432604:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6257.927071] LustreError: 432604:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764750839 with bad export cookie 7815164722730063345 [ 6257.931124] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6257.932108] LustreError: 432604:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6258.656227] LustreError: 432623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6258.656512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6258.661954] LustreError: 432623:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6258.666427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6258.671915] Lustre: Skipped 2 previous similar messages [ 6262.687965] LustreError: 432623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6262.688261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6262.693575] LustreError: 432623:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6263.982471] LustreError: 435095:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6263.985460] LustreError: 435095:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6264.027787] Lustre: server umount lustre-OST0000 complete [ 6266.420131] LustreError: 435448:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6266.422166] LustreError: 435448:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6266.485919] Lustre: server umount lustre-MDT0001 complete [ 6270.329393] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6271.389525] Key type lgssc unregistered [ 6271.530088] LNet: 435979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6271.534589] LNetError: 435979:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6271.542455] LNet: Removed LNI 192.168.201.137@tcp [ 6271.863692] Key type .llcrypt unregistered [ 6271.865550] Key type ._llcrypt unregistered [ 6280.909016] Key type ._llcrypt registered [ 6280.909865] Key type .llcrypt registered [ 6280.946043] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6281.366274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6281.376455] alg: No test for adler32 (adler32-zlib) [ 6282.222473] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 6282.308021] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6283.879138] Key type lgssc registered [ 6284.270534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6286.917930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6286.924980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6288.094947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6289.446278] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6292.262298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6292.395716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6293.727686] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6295.497732] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6296.677891] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6299.105380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6299.236893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6301.290830] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6303.133370] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6312.549508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6318.050758] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6318.051080] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6318.054178] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6318.057875] Lustre: Skipped 1 previous similar message [ 6318.061567] Lustre: Skipped 1 previous similar message [ 6318.958325] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [ 6320.767370] LustreError: 439619:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6320.793818] Lustre: server umount lustre-MDT0001 complete [ 6323.168083] LustreError: 437353:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6323.169103] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6323.169362] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6323.174987] LustreError: 437353:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6323.181338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6328.288191] LustreError: 437353:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6328.288668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6328.623161] LustreError: 439821:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6328.626260] LustreError: 439821:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6328.678275] Lustre: server umount lustre-OST0000 complete [ 6330.025592] LustreError: 440023:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6330.028556] LustreError: 440023:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6330.099304] Lustre: server umount lustre-MDT0000 complete [ 6334.659179] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6335.673881] Key type lgssc unregistered [ 6335.802435] LNet: 440707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6335.804505] LNetError: 440707:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6335.816272] LNet: Removed LNI 192.168.201.137@tcp [ 6336.113678] Key type .llcrypt unregistered [ 6336.114698] Key type ._llcrypt unregistered [ 6344.292209] Key type ._llcrypt registered [ 6344.293155] Key type .llcrypt registered [ 6344.333313] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 6350.324697] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6350.783360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6350.861831] alg: No test for adler32 (adler32-zlib) [ 6351.719994] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 6351.799080] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6353.383121] Key type lgssc registered [ 6353.770699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6356.724356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6358.886585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6360.909873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6363.029103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6366.314833] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6369.465884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6371.775183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6371.799530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6371.805679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6372.908855] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6372.923240] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6372.957656] Lustre: lustre-MDT0000: new disk, initializing [ 6372.981825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6372.988191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6374.337456] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6378.036947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6380.480766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6380.507118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6380.606446] Lustre: lustre-OST0000: new disk, initializing [ 6380.609087] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6380.611472] Lustre: Skipped 1 previous similar message [ 6380.639995] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6382.672523] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6385.825584] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6387.709941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6387.712859] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6387.722420] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6394.292903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6396.637875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6396.669409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6396.716289] Lustre: 444652:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6396.750932] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6396.787658] Lustre: lustre-MDT0001: new disk, initializing [ 6396.835595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6396.852568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6396.857369] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6398.268584] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6402.053425] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6402.136288] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6403.476726] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6405.588690] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 2 sec [ 6408.043072] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [ 6410.209257] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6410.210890] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6410.211939] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6410.214761] Lustre: Skipped 1 previous similar message [ 6410.220122] Lustre: Skipped 1 previous similar message [ 6413.155279] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.37@tcp (stopping) [ 6413.377939] LustreError: 448090:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6413.403250] Lustre: server umount lustre-MDT0001 complete [ 6420.449800] LustreError: 445666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6420.456426] LustreError: 445666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6425.570321] LustreError: 445666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6425.575748] LustreError: 445666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6428.639981] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6428.642572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6428.646720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6430.688853] LustreError: 446893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6431.917131] LustreError: 448361:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 6431.919196] LustreError: 448361:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6431.954974] Lustre: server umount lustre-OST0000 complete [ 6439.359042] LustreError: 448562:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6439.361698] LustreError: 448562:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6439.423686] Lustre: server umount lustre-MDT0000 complete [ 6443.927745] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6444.999766] Key type lgssc unregistered [ 6445.137353] LNet: 449246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6445.140942] LNetError: 449246:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6445.149476] LNet: Removed LNI 192.168.201.137@tcp [ 6445.490902] Key type .llcrypt unregistered [ 6445.492629] Key type ._llcrypt unregistered [ 6458.795469] Key type ._llcrypt registered [ 6458.796987] Key type .llcrypt registered [ 6458.840619] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6458.950228] Key type .llcrypt unregistered [ 6458.951192] Key type ._llcrypt unregistered [ 6465.981893] Key type ._llcrypt registered [ 6465.982888] Key type .llcrypt registered [ 6466.026601] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6466.317795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6466.325360] alg: No test for adler32 (adler32-zlib) [ 6467.191433] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 6467.270673] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6468.847151] Key type lgssc registered [ 6469.266111] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6472.129731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6474.427702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6476.657853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6479.053350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6479.075437] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6479.081902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6480.171703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6480.183562] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6480.221337] Lustre: lustre-MDT0000: new disk, initializing [ 6480.250244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6480.259271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6481.668065] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6485.953459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6485.985456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6486.026401] Lustre: 452565:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6486.050836] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6486.053868] Lustre: Skipped 1 previous similar message [ 6486.093815] Lustre: lustre-MDT0001: new disk, initializing [ 6486.131406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6486.145337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6486.149702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6487.537481] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6490.809929] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6492.104049] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6494.606183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6494.633202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6494.728386] Lustre: lustre-OST0000: new disk, initializing [ 6494.730341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6494.761150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6496.767232] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6497.342619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6497.346417] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6497.386773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6500.003811] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6502.369212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6502.371131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6502.375313] Lustre: Skipped 2 previous similar messages [ 6502.377458] Lustre: Skipped 1 previous similar message [ 6507.488610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6507.491607] Lustre: Skipped 3 previous similar messages [ 6508.478939] LustreError: 455007:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6508.504554] Lustre: server umount lustre-MDT0000 complete [ 6509.744159] LustreError: 452555:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764751091 with bad export cookie 1453621041377961711 [ 6509.747825] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6509.749670] LustreError: 452555:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6509.818600] LustreError: 455209:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6509.821576] LustreError: 455209:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6509.876910] Lustre: server umount lustre-MDT0001 complete [ 6521.394416] LustreError: 455409:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6521.397298] LustreError: 455409:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6521.440077] Lustre: server umount lustre-OST0000 complete [ 6525.050117] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 03:38:25 (1764751105) [ 6525.405173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6527.831303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6528.049956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6529.477259] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6532.524779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6532.665589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6534.149269] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6535.962991] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6537.223394] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6539.537110] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6539.564549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6539.635393] Lustre: Found index 0 for lustre-OST0000, updating log [ 6541.709151] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6543.512631] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6554.024800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6563.297855] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6563.300418] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6563.303995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6566.317733] LustreError: 458515:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6566.319777] LustreError: 458515:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6566.360743] Lustre: server umount lustre-OST0000 complete [ 6567.282317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6568.415808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6568.418835] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6568.422542] Lustre: Skipped 1 previous similar message [ 6568.424199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6568.426577] Lustre: Skipped 3 previous similar messages [ 6575.583994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6575.586859] Lustre: Skipped 5 previous similar messages [ 6582.239128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6582.272644] LustreError: 458771:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6582.275810] LustreError: 458771:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6582.303703] Lustre: server umount lustre-MDT0000 complete [ 6583.519042] LustreError: 456216:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764751165 with bad export cookie 1453621041377963104 [ 6583.522466] LustreError: 456216:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6583.522770] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6583.642622] Lustre: server umount lustre-MDT0001 complete [ 6586.416628] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 03:39:27 (1764751167) [ 6586.742376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6587.067547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6587.399463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6587.718587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6588.033582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6588.354442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6588.688291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6589.023396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6589.347234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6589.650915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6589.965461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6590.290542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6590.605843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6590.920435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6591.242091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6591.570632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6591.895377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6592.218921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6592.552533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6592.872670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6593.194639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6593.522053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6593.842585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6594.171356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6594.494929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6594.809177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6595.145191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6595.468176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6595.774052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6596.094577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6596.416661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6596.747985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6597.083558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6597.428857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6597.787240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6598.137565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6598.483544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6598.806056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6599.121621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6599.441775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6599.783493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6600.135823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6600.478390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6600.818073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6601.159610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6601.489512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6601.823478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6602.157317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6602.502797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6602.843773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6603.200390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6603.488243] LNet: 1 local NIs in recovery (showing 1): 192.168.201.137@tcp [ 6605.781953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6605.918090] LustreError: 462358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6605.925915] LustreError: 462358:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6605.942366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6605.944732] Lustre: Skipped 1 previous similar message [ 6607.239159] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6609.979397] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6611.394799] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6613.116550] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6614.266384] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6616.396596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6616.434063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6616.511676] Lustre: Found index 0 for lustre-OST0000, updating log [ 6617.637130] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 6618.340193] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6620.044342] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6622.688030] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6622.690031] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6622.692987] Lustre: Skipped 1 previous similar message [ 6622.694630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6622.697357] Lustre: Skipped 2 previous similar messages [ 6634.975120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6634.993158] LustreError: 464649:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6634.994938] LustreError: 464649:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6635.029044] Lustre: server umount lustre-OST0000 complete [ 6636.223079] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6636.226291] Lustre: Skipped 1 previous similar message [ 6639.328161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6639.329702] Lustre: Skipped 7 previous similar messages [ 6642.385880] Lustre: server umount lustre-MDT0000 complete [ 6642.401338] LustreError: 462376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6642.408554] LustreError: 462376:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6643.424960] LustreError: 462376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6643.429022] LustreError: 462376:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6643.555990] LustreError: 462383:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764751225 with bad export cookie 1453621041377964175 [ 6643.558618] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6643.560484] LustreError: 462383:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6643.657461] Lustre: server umount lustre-MDT0001 complete [ 6644.541157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6649.480320] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 6652.460554] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6655.699730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6657.812688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6659.694534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6661.798607] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6664.024259] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 03:40:44 (1764751244) [ 6667.999281] LNet: 1 local NIs in recovery (showing 1): 192.168.201.137@tcp [ 6676.714433] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6677.713350] Key type lgssc unregistered [ 6679.903600] LNet: 468624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6679.906358] LNetError: 468624:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6800.743309] LNet: Removed LNI 192.168.201.137@tcp [ 6801.088590] Key type .llcrypt unregistered [ 6801.090163] Key type ._llcrypt unregistered [ 6807.862122] Key type ._llcrypt registered [ 6807.863946] Key type .llcrypt registered [ 6807.903439] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6808.239865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6808.273071] alg: No test for adler32 (adler32-zlib) [ 6809.118175] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 6809.198535] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6810.783157] Key type lgssc registered [ 6811.190081] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6812.584887] LNet: Added LNI 192.168.201.137@tcp100 [8/256/0/180] [ 6812.889665] LNet: Added LNI 192.168.201.137@tcp101 [8/256/0/180] [ 6813.183167] LNet: Added LNI 192.168.201.137@tcp102 [8/256/0/180] [ 6813.551309] LNet: Added LNI 192.168.201.137@tcp103 [8/256/0/180] [ 6813.978819] LNet: Added LNI 192.168.201.137@tcp104 [8/256/0/180] [ 6814.396062] LNet: Added LNI 192.168.201.137@tcp105 [8/256/0/180] [ 6814.817206] LNet: Added LNI 192.168.201.137@tcp106 [8/256/0/180] [ 6815.270606] LNet: Added LNI 192.168.201.137@tcp107 [8/256/0/180] [ 6815.553438] LNet: Added LNI 192.168.201.137@tcp108 [8/256/0/180] [ 6815.813381] LNet: Added LNI 192.168.201.137@tcp109 [8/256/0/180] [ 6816.077498] LNet: Added LNI 192.168.201.137@tcp110 [8/256/0/180] [ 6816.441940] LNet: Added LNI 192.168.201.137@tcp111 [8/256/0/180] [ 6816.865670] LNet: Added LNI 192.168.201.137@tcp112 [8/256/0/180] [ 6817.284805] LNet: Added LNI 192.168.201.137@tcp113 [8/256/0/180] [ 6817.700369] LNet: Added LNI 192.168.201.137@tcp114 [8/256/0/180] [ 6818.057350] LNet: Added LNI 192.168.201.137@tcp115 [8/256/0/180] [ 6818.449906] LNet: Added LNI 192.168.201.137@tcp116 [8/256/0/180] [ 6818.818937] LNet: Added LNI 192.168.201.137@tcp117 [8/256/0/180] [ 6819.189970] LNet: Added LNI 192.168.201.137@tcp118 [8/256/0/180] [ 6819.542474] LNet: Added LNI 192.168.201.137@tcp119 [8/256/0/180] [ 6819.825060] LNet: Added LNI 192.168.201.137@tcp120 [8/256/0/180] [ 6820.115199] LNet: Added LNI 192.168.201.137@tcp121 [8/256/0/180] [ 6820.413638] LNet: Added LNI 192.168.201.137@tcp122 [8/256/0/180] [ 6820.770862] LNet: Added LNI 192.168.201.137@tcp123 [8/256/0/180] [ 6821.209166] LNet: Added LNI 192.168.201.137@tcp124 [8/256/0/180] [ 6821.649832] LNet: Added LNI 192.168.201.137@tcp125 [8/256/0/180] [ 6822.066519] LNet: Added LNI 192.168.201.137@tcp126 [8/256/0/180] [ 6822.496633] LNet: Added LNI 192.168.201.137@tcp127 [8/256/0/180] [ 6822.901709] LNet: Added LNI 192.168.201.137@tcp128 [8/256/0/180] [ 6823.293828] LNet: Added LNI 192.168.201.137@tcp129 [8/256/0/180] [ 6823.621620] LNet: Added LNI 192.168.201.137@tcp130 [8/256/0/180] [ 6824.027241] LNet: Added LNI 192.168.201.137@tcp131 [8/256/0/180] [ 6824.373185] LNet: Added LNI 192.168.201.137@tcp132 [8/256/0/180] [ 6824.714520] LNet: Added LNI 192.168.201.137@tcp133 [8/256/0/180] [ 6825.048921] LNet: Added LNI 192.168.201.137@tcp134 [8/256/0/180] [ 6825.507580] LNet: Added LNI 192.168.201.137@tcp135 [8/256/0/180] [ 6827.991790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6828.012186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6828.016866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6829.054161] Lustre: 471611:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.137@tcp: can't add NID 192.168.201.137@tcp130: rc = -75 [ 6829.139800] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6829.157563] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6829.205514] Lustre: lustre-MDT0000: new disk, initializing [ 6829.238988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6829.247108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6830.739554] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6835.339051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6835.363111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6835.409652] Lustre: 471630:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6835.449439] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6835.452751] Lustre: Skipped 1 previous similar message [ 6835.501850] Lustre: lustre-MDT0001: new disk, initializing [ 6835.516443] Lustre: 472415:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.137@tcp: can't add NID 192.168.201.137@tcp130: rc = -75 [ 6835.519099] Lustre: 472415:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6835.558957] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6835.569658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6835.574806] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6836.724065] Lustre: 472598:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.137@tcp: can't add NID 192.168.201.137@tcp130: rc = -75 [ 6836.727020] Lustre: 472598:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6836.993181] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6840.634207] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6842.260659] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6845.098059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6845.131051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6845.266354] Lustre: lustre-OST0000: new disk, initializing [ 6845.268678] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6845.318769] Lustre: 473385:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.137@tcp: can't add NID 192.168.201.137@tcp130: rc = -75 [ 6845.323991] Lustre: 473385:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 6845.354003] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6847.730265] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6851.120254] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6851.579743] Lustre: 473990:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.137@tcp: can't add NID 192.168.201.137@tcp130: rc = -75 [ 6851.583063] Lustre: 473990:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 6851.607974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6851.611128] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6851.663605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6856.675647] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6856.676392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6856.678050] LustreError: Skipped 1 previous similar message [ 6856.678636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6856.682467] Lustre: Skipped 1 previous similar message [ 6856.687906] Lustre: Skipped 1 previous similar message [ 6859.820716] LustreError: 474220:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6859.856574] Lustre: server umount lustre-OST0000 complete [ 6861.280067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6861.283293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6861.289442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6866.912877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6866.918046] Lustre: Skipped 2 previous similar messages [ 6867.334468] LustreError: 474423:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6867.340409] LustreError: 474423:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6867.381713] Lustre: server umount lustre-MDT0000 complete [ 6868.831535] LustreError: 473379:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764751450 with bad export cookie 16258618030326402021 [ 6868.835601] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6868.836608] LustreError: 473379:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6868.922708] LustreError: 474626:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6868.925117] LustreError: 474626:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6868.976447] Lustre: server umount lustre-MDT0001 complete [ 6872.608059] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6873.753724] Key type lgssc unregistered [ 6873.922890] LNet: 475165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6873.926140] LNetError: 475165:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6873.928414] LNet: Removed LNI 192.168.201.137@tcp [ 6874.975136] LNet: Removed LNI 192.168.201.137@tcp100 [ 6874.977493] LNetError: 475165:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6874.981323] LNetError: 475165:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6874.985995] LNet: Removed LNI 192.168.201.137@tcp101 [ 6875.999191] LNet: Removed LNI 192.168.201.137@tcp102 [ 6876.001944] LNetError: 475165:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6876.006639] LNetError: 475165:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 6876.011742] LNet: Removed LNI 192.168.201.137@tcp103 [ 6877.023197] LNet: Removed LNI 192.168.201.137@tcp104 [ 6878.047158] LNet: Removed LNI 192.168.201.137@tcp105 [ 6878.048537] LNetError: 475165:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6878.050555] LNetError: 475165:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 6878.052826] LNet: Removed LNI 192.168.201.137@tcp106 [ 6879.071192] LNet: Removed LNI 192.168.201.137@tcp107 [ 6880.095165] LNet: Removed LNI 192.168.201.137@tcp108 [ 6881.119304] LNet: Removed LNI 192.168.201.137@tcp109 [ 6882.143247] LNet: Removed LNI 192.168.201.137@tcp110 [ 6882.146859] LNetError: 475165:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6882.150831] LNetError: 475165:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 6882.156270] LNet: Removed LNI 192.168.201.137@tcp111 [ 6883.167134] LNet: Removed LNI 192.168.201.137@tcp112 [ 6884.191275] LNet: Removed LNI 192.168.201.137@tcp113 [ 6885.215248] LNet: Removed LNI 192.168.201.137@tcp114 [ 6886.239205] LNet: Removed LNI 192.168.201.137@tcp115 [ 6886.242957] LNet: Removed LNI 192.168.201.137@tcp116 [ 6886.245533] LNet: Removed LNI 192.168.201.137@tcp117 [ 6886.247995] LNet: Removed LNI 192.168.201.137@tcp118 [ 6886.250493] LNet: Removed LNI 192.168.201.137@tcp119 [ 6886.252889] LNet: Removed LNI 192.168.201.137@tcp120 [ 6886.255367] LNet: Removed LNI 192.168.201.137@tcp121 [ 6886.257657] LNet: Removed LNI 192.168.201.137@tcp122 [ 6886.259980] LNet: Removed LNI 192.168.201.137@tcp123 [ 6886.262593] LNet: Removed LNI 192.168.201.137@tcp124 [ 6886.265169] LNet: Removed LNI 192.168.201.137@tcp125 [ 6886.267391] LNet: Removed LNI 192.168.201.137@tcp126 [ 6886.269491] LNet: Removed LNI 192.168.201.137@tcp127 [ 6886.271593] LNet: Removed LNI 192.168.201.137@tcp128 [ 6886.273824] LNet: Removed LNI 192.168.201.137@tcp129 [ 6886.277035] LNet: Removed LNI 192.168.201.137@tcp130 [ 6886.278896] LNet: Removed LNI 192.168.201.137@tcp131 [ 6886.280732] LNet: Removed LNI 192.168.201.137@tcp132 [ 6886.282436] LNet: Removed LNI 192.168.201.137@tcp133 [ 6886.284119] LNet: Removed LNI 192.168.201.137@tcp134 [ 6886.291569] LNet: Removed LNI 192.168.201.137@tcp135 [ 6886.660640] Key type .llcrypt unregistered [ 6886.664090] Key type ._llcrypt unregistered [ 6894.628152] Key type ._llcrypt registered [ 6894.629126] Key type .llcrypt registered [ 6894.670413] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 6902.000391] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6902.431921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6902.521534] alg: No test for adler32 (adler32-zlib) [ 6903.369235] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 6903.444243] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6905.015190] Key type lgssc registered [ 6905.611919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6909.774909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6912.344445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6915.311058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6918.089283] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6920.711784] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 03:45:01 (1764751501) [ 6926.836696] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 6927.925269] Key type lgssc unregistered [ 6928.054430] LNet: 479177:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6928.056613] LNetError: 479177:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6928.068371] LNet: Removed LNI 192.168.201.137@tcp [ 6928.343455] Key type .llcrypt unregistered [ 6928.344485] Key type ._llcrypt unregistered [ 6935.303765] Key type ._llcrypt registered [ 6935.304888] Key type .llcrypt registered [ 6935.340769] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 6935.736656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6935.745834] alg: No test for adler32 (adler32-zlib) [ 6936.597020] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 6936.679816] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 6938.271227] Key type lgssc registered [ 6938.750159] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6942.295832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6942.312583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6942.318190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6943.404764] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6943.418907] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6943.470190] Lustre: lustre-MDT0000: new disk, initializing [ 6943.496146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6943.503524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6944.918935] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6949.817881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6949.840332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6949.872149] Lustre: 480349:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6949.895058] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6949.897390] Lustre: Skipped 1 previous similar message [ 6949.934566] Lustre: lustre-MDT0001: new disk, initializing [ 6949.964909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6949.976169] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6949.981529] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6951.448543] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6955.026260] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6956.795320] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6959.866046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6959.895966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6960.033614] Lustre: lustre-OST0000: new disk, initializing [ 6960.037754] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6960.088571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6960.523988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6960.534487] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6960.588874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6962.462512] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 6966.099826] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6967.491666] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing lnet_if_list [ 6969.225121] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 6969.236503] LNet: Added LNI 192.168.201.137@tcp10 [8/256/0/180] [ 6970.833213] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 6970.842555] LNet: Added LNI 192.168.201.137@tcp11 [8/256/0/180] [ 6972.028555] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 6972.038038] LNet: Added LNI 192.168.201.137@tcp12 [8/256/0/180] [ 6973.266535] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 6973.276364] LNet: Added LNI 192.168.201.137@tcp13 [8/256/0/180] [ 6974.676618] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 6974.685690] LNet: Added LNI 192.168.201.137@tcp14 [8/256/0/180] [ 6975.983637] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 6975.994522] LNet: Added LNI 192.168.201.137@tcp15 [8/256/0/180] [ 6977.352762] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 6977.366449] LNet: Added LNI 192.168.201.137@tcp16 [8/256/0/180] [ 6978.937145] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 6978.950168] LNet: Added LNI 192.168.201.137@tcp17 [8/256/0/180] [ 6980.486360] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 6980.498623] LNet: Added LNI 192.168.201.137@tcp18 [8/256/0/180] [ 6982.082985] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 6982.095703] LNet: Added LNI 192.168.201.137@tcp19 [8/256/0/180] [ 6983.716447] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 6983.727262] LNet: Added LNI 192.168.201.137@tcp20 [8/256/0/180] [ 6985.693122] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 6985.705905] LNet: Added LNI 192.168.201.137@tcp21 [8/256/0/180] [ 6987.447176] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 6987.457902] LNet: Added LNI 192.168.201.137@tcp22 [8/256/0/180] [ 6989.323547] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 6989.339424] LNet: Added LNI 192.168.201.137@tcp23 [8/256/0/180] [ 6991.505117] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 6991.518080] LNet: Added LNI 192.168.201.137@tcp24 [8/256/0/180] [ 6993.515252] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 6993.528381] LNet: Added LNI 192.168.201.137@tcp25 [8/256/0/180] [ 6995.540657] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 6995.559863] LNet: Added LNI 192.168.201.137@tcp26 [8/256/0/180] [ 6998.307448] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 6998.326748] LNet: Added LNI 192.168.201.137@tcp27 [8/256/0/180] [ 6998.338006] LNetError: 479608:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.137 [ 6998.344146] LNetError: Protocol error connecting to 192.168.201.137@tcp26 at host 192.168.201.137:988: is it running a compatible version of Lustre? [ 7000.362115] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7000.371981] LNet: Added LNI 192.168.201.137@tcp28 [8/256/0/180] [ 7000.375400] LNetError: 479609:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.137 [ 7000.380580] LNetError: Protocol error connecting to 192.168.201.137@tcp26 at host 192.168.201.137:988: is it running a compatible version of Lustre? [ 7002.235250] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7002.244615] LNet: Added LNI 192.168.201.137@tcp29 [8/256/0/180] [ 7002.248808] LNetError: 479611:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.137 [ 7002.252219] LNetError: Protocol error connecting to 192.168.201.137@tcp28 at host 192.168.201.137:988: is it running a compatible version of Lustre? [ 7003.231227] LNet: 1 local NIs in recovery (showing 1): 192.168.201.137@tcp28 [ 7004.130051] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7004.139872] LNet: Added LNI 192.168.201.137@tcp30 [8/256/0/180] [ 7005.280511] LNetError: 479610:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.137 [ 7005.284171] LNetError: 479610:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [ 7005.286818] LNetError: Protocol error connecting to 192.168.201.137@tcp28 at host 192.168.201.137:988: is it running a compatible version of Lustre? [ 7005.290644] LNetError: Skipped 1 previous similar message [ 7006.033819] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7006.042892] LNet: Added LNI 192.168.201.137@tcp31 [8/256/0/180] [ 7008.151778] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7008.165187] LNet: Added LNI 192.168.201.137@tcp32 [8/256/0/180] [ 7009.378797] LNetError: 479608:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.137 [ 7009.382578] LNetError: 479608:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 5 previous similar messages [ 7009.385280] LNetError: Protocol error connecting to 192.168.201.137@tcp28 at host 192.168.201.137:988: is it running a compatible version of Lustre? [ 7009.388521] LNetError: Skipped 6 previous similar messages [ 7010.640803] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7010.655766] LNet: Added LNI 192.168.201.137@tcp33 [8/256/0/180] [ 7013.172587] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7013.184454] LNet: Added LNI 192.168.201.137@tcp34 [8/256/0/180] [ 7015.516135] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7015.529610] LNet: Added LNI 192.168.201.137@tcp35 [8/256/0/180] [ 7017.568151] LNetError: 479608:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.137 [ 7017.571028] LNetError: 479608:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 8 previous similar messages [ 7017.573181] LNetError: Protocol error connecting to 192.168.201.137@tcp28 at host 192.168.201.137:988: is it running a compatible version of Lustre? [ 7017.576080] LNetError: Skipped 5 previous similar messages [ 7017.807622] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7017.823499] LNet: Added LNI 192.168.201.137@tcp36 [8/256/0/180] [ 7018.987958] LNetError: 479618:0:(peer.c:1877:lnet_peer_add_nid()) Peer 192.168.201.137@tcp NID 192.168.201.137@tcp is already locked with peer 192.168.201.137@tcp [ 7020.611180] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7020.625603] LNet: Added LNI 192.168.201.137@tcp37 [8/256/0/180] [ 7023.261177] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7023.280624] LNet: Added LNI 192.168.201.137@tcp38 [8/256/0/180] [ 7026.512198] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7026.530028] LNet: Added LNI 192.168.201.137@tcp39 [8/256/0/180] [ 7029.092666] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7029.110075] LNet: Added LNI 192.168.201.137@tcp40 [8/256/0/180] [ 7032.236942] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7032.247226] LNet: Added LNI 192.168.201.137@tcp41 [8/256/0/180] [ 7033.957402] LNetError: 492653:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.137 [ 7033.967856] LNetError: 492653:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 30 previous similar messages [ 7033.979306] LNetError: Protocol error connecting to 192.168.201.137@tcp38 at host 192.168.201.137:988: is it running a compatible version of Lustre? [ 7033.989731] LNetError: Skipped 30 previous similar messages [ 7034.879455] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7034.891515] LNet: Added LNI 192.168.201.137@tcp42 [8/256/0/180] [ 7037.629941] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7037.640610] LNet: Added LNI 192.168.201.137@tcp43 [8/256/0/180] [ 7040.117665] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7040.127224] LNet: Added LNI 192.168.201.137@tcp44 [8/256/0/180] [ 7042.794626] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7042.815180] LNet: Added LNI 192.168.201.137@tcp45 [8/256/0/180] [ 7045.667965] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7045.684444] LNet: Added LNI 192.168.201.137@tcp46 [8/256/0/180] [ 7048.686278] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7048.702516] LNet: Added LNI 192.168.201.137@tcp47 [8/256/0/180] [ 7052.098214] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7052.118427] LNet: Added LNI 192.168.201.137@tcp48 [8/256/0/180] [ 7055.488463] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7055.504877] LNet: Added LNI 192.168.201.137@tcp49 [8/256/0/180] [ 7058.640768] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7058.654303] LNet: Added LNI 192.168.201.137@tcp50 [8/256/0/180] [ 7062.623508] LNet: 9 local NIs in recovery (showing 9): 192.168.201.137@tcp47, 192.168.201.137@tcp44, 192.168.201.137@tcp45, 192.168.201.137@tcp46, 192.168.201.137@tcp43, 192.168.201.137@tcp42, 192.168.201.137@tcp37, 192.168.201.137@tcp34, 192.168.201.137@tcp32 [ 7066.087270] LNetError: 479608:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.137 [ 7066.092607] LNetError: 479608:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 43 previous similar messages [ 7066.096620] LNetError: Protocol error connecting to 192.168.201.137@tcp49 at host 192.168.201.137:988: is it running a compatible version of Lustre? [ 7066.101985] LNetError: Skipped 43 previous similar messages [ 7068.131642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7068.132896] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7068.137019] LustreError: Skipped 1 previous similar message [ 7068.146956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7068.153842] Lustre: Skipped 1 previous similar message [ 7073.253373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7073.257738] Lustre: Skipped 1 previous similar message [ 7073.275157] LustreError: 504539:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7073.336189] Lustre: server umount lustre-OST0000 complete [ 7077.856436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7077.861685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7077.870350] Lustre: Skipped 1 previous similar message [ 7077.873874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7081.045471] LustreError: 504740:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7081.048868] LustreError: 504740:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7081.085035] Lustre: server umount lustre-MDT0000 complete [ 7082.321169] LustreError: 480339:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764751663 with bad export cookie 18123510687230968979 [ 7082.324345] LustreError: 480339:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7082.325299] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7082.400337] LustreError: 504945:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7082.402679] LustreError: 504945:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7082.455756] Lustre: server umount lustre-MDT0001 complete [ 7086.446615] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 7087.570706] Key type lgssc unregistered [ 7087.719389] LNet: 505484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7087.724678] LNetError: 505484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7087.729253] LNet: Removed LNI 192.168.201.137@tcp [ 7088.735193] LNet: Removed LNI 192.168.201.137@tcp10 [ 7088.737244] LNetError: 505484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7088.740293] LNetError: 505484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7088.743787] LNet: Removed LNI 192.168.201.137@tcp11 [ 7089.759212] LNet: Removed LNI 192.168.201.137@tcp12 [ 7089.763582] LNetError: 505484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7089.769383] LNetError: 505484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7089.774921] LNet: Removed LNI 192.168.201.137@tcp13 [ 7090.783264] LNet: Removed LNI 192.168.201.137@tcp14 [ 7091.807239] LNet: Removed LNI 192.168.201.137@tcp15 [ 7091.810088] LNetError: 505484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7091.815244] LNetError: 505484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7091.820599] LNet: Removed LNI 192.168.201.137@tcp16 [ 7092.831224] LNet: Removed LNI 192.168.201.137@tcp17 [ 7093.855220] LNet: Removed LNI 192.168.201.137@tcp18 [ 7094.879225] LNet: Removed LNI 192.168.201.137@tcp19 [ 7095.903213] LNet: Removed LNI 192.168.201.137@tcp20 [ 7095.905983] LNetError: 505484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7095.910799] LNetError: 505484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7095.916361] LNet: Removed LNI 192.168.201.137@tcp21 [ 7096.927315] LNet: Removed LNI 192.168.201.137@tcp22 [ 7097.951264] LNet: Removed LNI 192.168.201.137@tcp23 [ 7098.975272] LNet: Removed LNI 192.168.201.137@tcp24 [ 7099.999275] LNet: Removed LNI 192.168.201.137@tcp25 [ 7101.023132] LNet: Removed LNI 192.168.201.137@tcp26 [ 7102.047255] LNet: Removed LNI 192.168.201.137@tcp27 [ 7103.071197] LNet: Removed LNI 192.168.201.137@tcp28 [ 7104.095104] LNet: Removed LNI 192.168.201.137@tcp29 [ 7104.096321] LNetError: 505484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7104.098100] LNetError: 505484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 7104.100465] LNet: Removed LNI 192.168.201.137@tcp30 [ 7105.119271] LNet: Removed LNI 192.168.201.137@tcp31 [ 7106.143244] LNet: Removed LNI 192.168.201.137@tcp32 [ 7107.167212] LNet: Removed LNI 192.168.201.137@tcp33 [ 7107.170464] LNet: Removed LNI 192.168.201.137@tcp34 [ 7108.191226] LNet: Removed LNI 192.168.201.137@tcp35 [ 7109.215284] LNet: Removed LNI 192.168.201.137@tcp36 [ 7109.217912] LNet: Removed LNI 192.168.201.137@tcp37 [ 7110.239179] LNet: Removed LNI 192.168.201.137@tcp38 [ 7111.263229] LNet: Removed LNI 192.168.201.137@tcp39 [ 7112.287113] LNet: Removed LNI 192.168.201.137@tcp40 [ 7113.311276] LNet: Removed LNI 192.168.201.137@tcp41 [ 7113.314812] LNet: Removed LNI 192.168.201.137@tcp42 [ 7114.335234] LNet: Removed LNI 192.168.201.137@tcp43 [ 7114.337706] LNet: Removed LNI 192.168.201.137@tcp44 [ 7114.339895] LNet: Removed LNI 192.168.201.137@tcp45 [ 7114.342252] LNet: Removed LNI 192.168.201.137@tcp46 [ 7114.344622] LNet: Removed LNI 192.168.201.137@tcp47 [ 7115.359206] LNet: Removed LNI 192.168.201.137@tcp48 [ 7116.383093] LNet: Removed LNI 192.168.201.137@tcp49 [ 7116.390381] LNet: Removed LNI 192.168.201.137@tcp50 [ 7116.658610] LNetError: 505546:0:(module.c:692:libcfs_exit()) Portals memory leaked: 2632 bytes [ 7116.690938] Key type .llcrypt unregistered [ 7116.692549] Key type ._llcrypt unregistered [ 7124.823335] Key type ._llcrypt registered [ 7124.824490] Key type .llcrypt registered [ 7124.859611] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7125.154718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7125.205112] alg: No test for adler32 (adler32-zlib) [ 7126.055840] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 7126.135234] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 7127.711197] Key type lgssc registered [ 7128.049244] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7130.962718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7130.973118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7132.135641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7133.551250] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7137.124464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7137.297538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7138.676786] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7141.017994] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7142.487928] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7145.006178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7145.202553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7147.389058] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7149.349120] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7163.511534] loop0: detected capacity change from 0 to 204800000 [ 7163.633089] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7163.639526] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7163.650944] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7163.683511] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7166.232566] loop0: detected capacity change from 0 to 204800000 [ 7166.288147] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7166.291544] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7166.294995] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7166.343329] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7168.152986] loop0: detected capacity change from 0 to 204800000 [ 7170.401090] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7170.423379] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7170.469088] Lustre: fs15246-OST0000: new disk, initializing [ 7170.470852] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7170.487702] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7172.375625] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7176.286436] loop1: detected capacity change from 0 to 204800000 [ 7178.420342] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7178.441330] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7178.474729] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7178.498532] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7178.532195] Lustre: fs15246-MDT0000: new disk, initializing [ 7178.576399] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7178.584093] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7178.622996] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7179.679744] Lustre: 511105:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000002634f86b x1850476285950848/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1764751777 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7180.018849] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7188.463876] LustreError: 511105:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 7188.469541] LustreError: 511105:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 7188.474683] LustreError: 511105:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 7193.061073] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7193.065544] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7194.639137] LustreError: 512230:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 7194.675334] Lustre: server umount fs15246-MDT0000 complete [ 7198.183331] LustreError: 506701:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7198.194373] LustreError: 506701:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7198.490930] LustreError: 512734:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 7198.494871] LustreError: 512734:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7198.517479] Lustre: server umount fs15246-OST0000 complete [ 7203.301139] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7203.303506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7203.308061] Lustre: Skipped 1 previous similar message [ 7208.420920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7208.424768] Lustre: Skipped 2 previous similar messages [ 7208.493096] LustreError: 513235:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7208.494842] LustreError: 513235:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7208.525456] Lustre: server umount lustre-OST0000 complete [ 7213.538297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7213.539609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7213.547283] Lustre: Skipped 1 previous similar message [ 7216.323463] LustreError: 513437:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7216.326276] LustreError: 513437:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7216.359083] Lustre: server umount lustre-MDT0000 complete [ 7217.670670] LustreError: 506689:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764751799 with bad export cookie 14461473294542864367 [ 7217.676936] LustreError: 506689:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7217.677589] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7217.826614] Lustre: server umount lustre-MDT0001 complete [ 7221.828207] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 7222.767852] Key type lgssc unregistered [ 7222.887355] LNet: 514169:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7222.889577] LNetError: 514169:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7222.898302] LNet: Removed LNI 192.168.201.137@tcp [ 7223.241874] Key type .llcrypt unregistered [ 7223.243635] Key type ._llcrypt unregistered [ 7230.016258] Key type ._llcrypt registered [ 7230.017214] Key type .llcrypt registered [ 7230.048362] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7230.446961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7230.452905] alg: No test for adler32 (adler32-zlib) [ 7231.298431] Lustre: Lustre: Build Version: 2.17.0_RC1_2_g29d2772 [ 7231.377665] LNet: Added LNI 192.168.201.137@tcp [8/256/0/180] [ 7232.959252] Key type lgssc registered [ 7233.352790] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7235.828664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7237.928223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7240.190519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7241.872726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7247.644664] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 7250.989244] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7254.636393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7256.961588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7259.266654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7262.067400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7265.314073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7265.337084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7265.341997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7266.430191] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7266.442469] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7266.482838] Lustre: lustre-MDT0000: new disk, initializing [ 7266.506987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7266.514197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7268.036304] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7272.428631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7272.454391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7272.497626] Lustre: 518572:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7272.542257] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7272.545470] Lustre: Skipped 1 previous similar message [ 7272.587081] Lustre: lustre-MDT0001: new disk, initializing [ 7272.632064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7272.652116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7272.656572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7274.344183] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7277.580220] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7278.965822] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7281.524715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7281.553174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7281.692236] Lustre: lustre-OST0000: new disk, initializing [ 7281.696063] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7281.739721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7283.646476] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_default_debug -1 all [ 7285.752055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7285.755226] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7285.784884] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7287.057429] Lustre: DEBUG MARKER: oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7288.571595] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7288.633038] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7290.004816] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7290.081309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7295.971870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7295.972958] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7295.977498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7295.983122] Lustre: Skipped 1 previous similar message [ 7301.089859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7301.091826] Lustre: Skipped 2 previous similar messages [ 7305.183097] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7305.202620] LustreError: 521513:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7305.236362] Lustre: server umount lustre-OST0000 complete [ 7308.768183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7308.772491] LustreError: Skipped 1 previous similar message [ 7308.774990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7308.782754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7311.328407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7311.329050] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7312.764892] LustreError: 521715:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7312.766880] LustreError: 521715:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7312.795815] Lustre: server umount lustre-MDT0000 complete [ 7313.998129] LustreError: 518562:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1764751895 with bad export cookie 8413958583639276893 [ 7314.001467] LustreError: MGC192.168.201.137@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7314.004444] LustreError: 518562:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7314.062516] LustreError: 521916:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7314.064332] LustreError: 521916:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7314.113423] Lustre: server umount lustre-MDT0001 complete [ 7323.301718] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing set_hostid [ 7326.231993] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing load_modules_local [ 7329.501573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7332.152293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7334.265249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7336.681524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7337.273516] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7195 sec ========== 03:51:58 (1764751918) [ 7337.832443] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 03:51:58 (1764751918) === [ 7338.918772] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 03:51:59 (1764751919) === [ 7351.584096] Lustre: DEBUG MARKER: oleg137-server.virtnet: executing unload_modules_local [ 7352.544896] Key type lgssc unregistered [ 7352.658350] LNet: 526764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7352.661050] LNetError: 526764:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7352.667292] LNet: Removed LNI 192.168.201.137@tcp [ 7352.936042] Key type .llcrypt unregistered [ 7352.937103] Key type ._llcrypt unregistered