[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 431158510 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2432MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002005] Switched APIC routing to physical x2apic. [ 0.003006] kvm-guest: setup PV IPIs [ 0.005478] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006011] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007004] pid_max: default: 32768 minimum: 301 [ 0.008098] LSM: Security Framework initializing [ 0.009028] Yama: becoming mindful. [ 0.010021] SELinux: Initializing. [ 0.011054] *** VALIDATE selinux *** [ 0.018161] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022695] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023104] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024071] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025070] *** VALIDATE tmpfs *** [ 0.026384] *** VALIDATE proc *** [ 0.027169] *** VALIDATE cgroup *** [ 0.028004] *** VALIDATE cgroup2 *** [ 0.029196] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.030111] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.031003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.032017] Spectre V2 : User space: Vulnerable [ 0.033004] Speculative Store Bypass: Vulnerable [ 0.036527] debug: unmapping init [mem 0xffffffffaf259000-0xffffffffaf260fff] [ 0.038261] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.039546] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.040011] ... version: 2 [ 0.040969] ... bit width: 48 [ 0.041006] ... generic registers: 4 [ 0.041939] ... value mask: 0000ffffffffffff [ 0.042006] ... max period: 00007fffffffffff [ 0.043006] ... fixed-purpose events: 3 [ 0.043949] ... event mask: 000000070000000f [ 0.044221] rcu: Hierarchical SRCU implementation. [ 0.046424] smp: Bringing up secondary CPUs ... [ 0.047416] x86: Booting SMP configuration: [ 0.048010] .... node #0, CPUs: #1 #2 #3 [ 0.057095] smp: Brought up 1 node, 4 CPUs [ 0.059012] smpboot: Max logical packages: 1 [ 0.060016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138703] node 0 deferred pages initialised in 76ms [ 0.142094] devtmpfs: initialized [ 0.143249] x86/mm: Memory block size: 128MB [ 0.153000] gcov: version magic: 0x41383552 [ 0.160048] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161240] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.163205] pinctrl core: initialized pinctrl subsystem [ 0.165109] [ 0.165469] ************************************************************* [ 0.166006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167007] ** ** [ 0.169007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.170006] ** ** [ 0.171006] ** This means that this kernel is built to expose internal ** [ 0.173006] ** IOMMU data structures, which may compromise security on ** [ 0.175008] ** your system. ** [ 0.176006] ** ** [ 0.177000] ** If you see this message and you are not debugging the ** [ 0.177000] ** kernel, report this immediately to your vendor! ** [ 0.178011] ** ** [ 0.179006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.181009] ************************************************************* [ 0.182660] NET: Registered protocol family 16 [ 0.183384] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.184049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.186036] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.189435] cpuidle: using governor menu [ 0.191011] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.193356] PCI: Using configuration type 1 for base access [ 0.195104] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.202107] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.204049] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.210286] cryptd: max_cpu_qlen set to 1000 [ 0.215652] ACPI: Added _OSI(Module Device) [ 0.218015] ACPI: Added _OSI(Processor Device) [ 0.219007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.221008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.226211] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.233148] ACPI: Interpreter enabled [ 0.234000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.234000] ACPI: Using IOAPIC for interrupt routing [ 0.236084] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.239541] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.254480] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.258038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.260010] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.264058] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.269628] acpiphp: Slot [2] registered [ 0.271070] acpiphp: Slot [5] registered [ 0.272506] acpiphp: Slot [6] registered [ 0.273066] acpiphp: Slot [7] registered [ 0.274000] acpiphp: Slot [8] registered [ 0.274000] acpiphp: Slot [9] registered [ 0.275063] acpiphp: Slot [10] registered [ 0.276063] acpiphp: Slot [3] registered [ 0.278073] acpiphp: Slot [4] registered [ 0.279061] acpiphp: Slot [11] registered [ 0.280104] acpiphp: Slot [12] registered [ 0.281102] acpiphp: Slot [13] registered [ 0.283066] acpiphp: Slot [14] registered [ 0.284052] acpiphp: Slot [15] registered [ 0.284880] acpiphp: Slot [16] registered [ 0.286052] acpiphp: Slot [17] registered [ 0.287081] acpiphp: Slot [18] registered [ 0.288075] acpiphp: Slot [19] registered [ 0.289048] acpiphp: Slot [20] registered [ 0.289856] acpiphp: Slot [21] registered [ 0.290058] acpiphp: Slot [22] registered [ 0.291046] acpiphp: Slot [23] registered [ 0.292065] acpiphp: Slot [24] registered [ 0.294064] acpiphp: Slot [25] registered [ 0.295086] acpiphp: Slot [26] registered [ 0.296044] acpiphp: Slot [27] registered [ 0.297063] acpiphp: Slot [28] registered [ 0.298059] acpiphp: Slot [29] registered [ 0.299057] acpiphp: Slot [30] registered [ 0.301075] acpiphp: Slot [31] registered [ 0.302000] PCI host bridge to bus 0000:00 [ 0.302000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.304016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.306011] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.309013] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.311010] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.314014] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.316161] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.318975] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.322000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.329820] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.334051] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.336018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.338019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.340019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.341588] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.344597] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.345032] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.349854] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.353000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.360740] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.362000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.364809] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.372013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.376011] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.390013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.398502] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.406015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.412012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.444016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.457360] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.475018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.490016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.514015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.527723] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.541018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.552016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.582017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.594012] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.608014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.612011] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.624015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.632371] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.641015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.645010] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.659020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.669101] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.670178] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.671000] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.671000] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.672124] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.676062] iommu: Default domain type: Passthrough [ 0.677254] SCSI subsystem initialized [ 0.678199] ACPI: bus type USB registered [ 0.679070] usbcore: registered new interface driver usbfs [ 0.681061] usbcore: registered new interface driver hub [ 0.683120] usbcore: registered new device driver usb [ 0.684150] pps_core: LinuxPPS API ver. 1 registered [ 0.686011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.689063] PTP clock support registered [ 0.691182] EDAC MC: Ver: 3.0.0 [ 0.694146] PCI: Using ACPI for IRQ routing [ 0.696852] NetLabel: Initializing [ 0.698023] NetLabel: domain hash size = 128 [ 0.700010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.703095] NetLabel: unlabeled traffic allowed by default [ 0.704294] vgaarb: loaded [ 0.705418] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.708016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.718996] clocksource: Switched to clocksource kvm-clock [ 0.810928] VFS: Disk quotas dquot_6.6.0 [ 0.812168] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.813720] *** VALIDATE ramfs *** [ 0.814739] *** VALIDATE hugetlbfs *** [ 0.815843] pnp: PnP ACPI init [ 0.817482] pnp: PnP ACPI: found 6 devices [ 0.832715] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.836367] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.838238] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.839567] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.841195] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.843078] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.844694] NET: Registered protocol family 2 [ 0.846570] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.850358] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.853249] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.858106] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.860563] TCP: Hash tables configured (established 65536 bind 65536) [ 0.863748] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.867118] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.869591] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.871568] NET: Registered protocol family 1 [ 0.874222] RPC: Registered named UNIX socket transport module. [ 0.876242] RPC: Registered udp transport module. [ 0.877483] RPC: Registered tcp transport module. [ 0.878770] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.881215] NET: Registered protocol family 44 [ 0.882572] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.884484] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.887145] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.888570] PCI: CLS 0 bytes, default 64 [ 0.889949] Unpacking initramfs... [ 2.519753] debug: unmapping init [mem 0xffff903efcc54000-0xffff903efffbffff] [ 2.523268] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.524890] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.527317] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.013943] Initialise system trusted keyrings [ 3.015565] Key type blacklist registered [ 3.017315] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.026405] zbud: loaded [ 3.029271] *** VALIDATE nfs *** [ 3.030419] *** VALIDATE nfs4 *** [ 3.031773] pstore: using deflate compression [ 3.035395] Platform Keyring initialized [ 3.146327] NET: Registered protocol family 38 [ 3.148167] Key type asymmetric registered [ 3.149558] Asymmetric key parser 'x509' registered [ 3.151499] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.154436] io scheduler mq-deadline registered [ 3.156143] io scheduler kyber registered [ 3.157846] io scheduler bfq registered [ 3.159540] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.162663] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.165582] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.168496] ACPI: Power Button [PWRF] [ 3.174413] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.181758] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.195167] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.201361] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.213456] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.242556] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.270763] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.275108] Non-volatile memory driver v1.3 [ 3.276557] Linux agpgart interface v0.103 [ 3.303495] virtio_blk virtio1: [vda] 134152 512-byte logical blocks (68.7 MB/65.5 MiB) [ 3.306151] vda: detected capacity change from 0 to 68685824 [ 3.322219] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.324440] vdb: detected capacity change from 0 to 1073741824 [ 3.340569] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.343258] vdc: detected capacity change from 0 to 2621440000 [ 3.360553] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.362946] vdd: detected capacity change from 0 to 2621440000 [ 3.389800] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.392113] vde: detected capacity change from 0 to 4294967296 [ 3.413093] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.415170] vdf: detected capacity change from 0 to 4294967296 [ 3.427667] libphy: Fixed MDIO Bus: probed [ 3.439747] usbcore: registered new interface driver usbserial_generic [ 3.441522] usbserial: USB Serial support registered for generic [ 3.443184] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.446444] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.447864] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.451272] mousedev: PS/2 mouse device common for all mice [ 3.453466] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.456374] rtc_cmos 00:05: RTC can wake from S4 [ 3.459843] rtc_cmos 00:05: registered as rtc0 [ 3.462664] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.462821] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.466536] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.467794] intel_pstate: CPU model not supported [ 3.473946] hid: raw HID events driver (C) Jiri Kosina [ 3.475443] usbcore: registered new interface driver usbhid [ 3.476894] usbhid: USB HID core driver [ 3.477974] drop_monitor: Initializing network drop monitor service [ 3.479657] Initializing XFRM netlink socket [ 3.481127] NET: Registered protocol family 10 [ 3.483327] Segment Routing with IPv6 [ 3.484224] NET: Registered protocol family 17 [ 3.485711] mpls_gso: MPLS GSO support [ 3.490740] RAS: Correctable Errors collector initialized. [ 3.492162] AVX version of gcm_enc/dec engaged. [ 3.493285] AES CTR mode by8 optimization enabled [ 3.558862] sched_clock: Marking stable (3558793797, 0)->(4364383283, -805589486) [ 3.561665] registered taskstats version 1 [ 3.562671] Loading compiled-in X.509 certificates [ 3.563663] zswap: loaded using pool lzo/zbud [ 3.576866] Key type big_key registered [ 3.583788] Key type encrypted registered [ 3.584580] ima: No TPM chip found, activating TPM-bypass! [ 3.585615] ima: Allocated hash algorithm: sha1 [ 3.586522] ima: No architecture policies found [ 3.587415] evm: Initialising EVM extended attributes: [ 3.588314] evm: security.selinux [ 3.588901] evm: security.ima [ 3.589448] evm: security.capability [ 3.590110] evm: HMAC attrs: 0x1 [ 3.592117] rtc_cmos 00:05: setting system clock to 2026-01-11 13:59:46 UTC (1768139986) [ 3.598869] debug: unmapping init [mem 0xffffffffb0203000-0xffffffffb03fffff] [ 3.601333] debug: unmapping init [mem 0xffffffffaef82000-0xffffffffaf258fff] [ 3.607088] Write protecting the kernel read-only data: 28672k [ 3.610450] debug: unmapping init [mem 0xffffffffad603000-0xffffffffad7fffff] [ 3.613112] debug: unmapping init [mem 0xffffffffadf14000-0xffffffffadffffff] [ 3.647321] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.655254] systemd[1]: Detected virtualization kvm. [ 3.656980] systemd[1]: Detected architecture x86-64. [ 3.658247] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.679757] systemd[1]: No hostname configured. [ 3.681675] systemd[1]: Set hostname to . [ 3.683244] random: systemd: uninitialized urandom read (16 bytes read) [ 3.685329] systemd[1]: Initializing machine ID from random generator. [ 3.787032] random: ln: uninitialized urandom read (6 bytes read) [ 3.877459] random: systemd: uninitialized urandom read (16 bytes read) [ 3.880177] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.884486] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.887938] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.576706] device-mapper: uevent: version 1.0.3 [ 4.578413] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.254468] virtio_net virtio0 ens2: renamed from eth0 [ 5.288907] random: fast init done [ 5.589139] scsi host0: ata_piix [ 5.650458] scsi host1: ata_piix [ 5.652457] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.657377] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.097747] random: crng init done [ 10.099195] random: 7 urandom warning(s) missed due to ratelimiting [ 10.811408] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.506501] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.903569] printk: systemd: 26 output lines suppressed due to ratelimiting [ 13.259337] SELinux: Disabled at runtime. [ 13.329848] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.337601] systemd[1]: Detected virtualization kvm. [ 13.339183] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.054951] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.058718] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.071622] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.076941] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.084577] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.091401] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.098257] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. [ 14.306719] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Mounted /mnt. [ 14.924700] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 15.477601] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.554170] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.764704] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 15.831908] EDAC sbridge: Ver: 1.1.2 [ 17.810256] Key type dns_resolver registered [ 18.193911] NFS: Registering the id_resolver key type [ 18.196227] Key type id_resolver registered [ 18.197867] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg304-server login: [ 41.470929] libcfs: loading out-of-tree module taints kernel. [ 41.485857] Key type ._llcrypt registered [ 41.486609] Key type .llcrypt registered [ 41.521938] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 48.657046] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 49.234876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 49.241990] alg: No test for adler32 (adler32-zlib) [ 50.213120] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 50.487065] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 52.087209] Key type lgssc registered [ 52.697795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 59.045979] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 72.524106] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 76.458522] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 76.465515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 77.543500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 77.555151] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 77.592995] Lustre: lustre-MDT0000: new disk, initializing [ 77.618320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 77.624949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 78.785235] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 83.754506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 83.799235] Lustre: 6464:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 83.812767] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 83.814405] Lustre: Skipped 1 previous similar message [ 83.854513] Lustre: lustre-MDT0001: new disk, initializing [ 83.872303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 83.878988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 83.882459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 85.174799] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 87.459785] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 90.622588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 90.711554] Lustre: lustre-OST0000: new disk, initializing [ 90.713214] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 90.727439] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 92.522625] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 92.526086] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 92.550367] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 92.608252] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 98.153683] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 98.195824] Lustre: lustre-OST0001: new disk, initializing [ 98.197522] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 98.216772] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 100.033379] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 100.460367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 100.463067] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 100.486628] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 105.870348] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 111.463521] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 118.398597] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing check_logdir /tmp/testlogs/ [ 119.834384] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing yml_node [ 121.229868] Lustre: DEBUG MARKER: Client: 2.17.50.31 [ 122.021079] Lustre: DEBUG MARKER: MDS: 2.17.50.31 [ 122.803984] Lustre: DEBUG MARKER: OSS: 2.17.50.31 [ 123.307461] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Jan 11 09:01:45 EST 2026 [ 128.970668] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 129.483363] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 141.791968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 141.792670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 141.797666] Lustre: Skipped 2 previous similar messages [ 141.799423] Lustre: Skipped 1 previous similar message [ 146.912275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 146.914801] Lustre: Skipped 3 previous similar messages [ 147.455349] LustreError: 12168:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 147.493325] Lustre: server umount lustre-MDT0000 complete [ 148.771558] LustreError: 6455:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768140131 with bad export cookie 6743050382008244824 [ 148.773204] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 148.775301] LustreError: 6455:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 152.031720] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 152.032565] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 152.036189] Lustre: Skipped 2 previous similar messages [ 152.039101] Lustre: Skipped 1 previous similar message [ 158.175445] LustreError: 6470:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 158.175506] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 158.179922] LustreError: 6470:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 6 previous similar messages [ 158.181709] Lustre: Skipped 3 previous similar messages [ 163.295120] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 163.295499] LustreError: 6470:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 163.295504] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 163.304683] LustreError: 6470:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 1 previous similar message [ 163.333821] LustreError: 12370:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 163.335895] LustreError: 12370:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 163.374854] Lustre: server umount lustre-MDT0001 complete [ 164.604070] LustreError: 12573:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 164.606801] LustreError: 12573:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 164.624192] Lustre: server umount lustre-OST0000 complete [ 165.929856] Lustre: server umount lustre-OST0001 complete [ 168.041953] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 170.859468] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 182.632200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.730742] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 182.743746] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 182.780849] Lustre: lustre-MDT0000: new disk, initializing [ 182.825416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 182.834808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 184.171227] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 188.207658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 188.249175] Lustre: 14940:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 188.348951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 189.593016] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 192.929545] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 194.938459] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 198.796737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 198.972814] Lustre: lustre-OST0000: new disk, initializing [ 198.980049] Lustre: Skipped 1 previous similar message [ 198.986502] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 198.994092] Lustre: Skipped 2 previous similar messages [ 199.057735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 199.060611] Lustre: Skipped 1 previous similar message [ 200.918316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 200.922525] Lustre: Skipped 1 previous similar message [ 200.925550] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 200.993538] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 202.631198] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 206.968877] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 209.379604] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 209.501028] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 211.938194] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 212.092289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 216.546189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 216.547590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 216.551768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 216.551779] Lustre: Skipped 1 previous similar message [ 216.556040] Lustre: Skipped 1 previous similar message [ 219.264208] LustreError: 17870:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 219.267190] LustreError: 17870:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 219.324225] Lustre: server umount lustre-OST0000 complete [ 221.664749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 221.669532] Lustre: Skipped 1 previous similar message [ 227.398955] LustreError: 18071:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 227.401368] LustreError: 18071:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 227.442173] Lustre: server umount lustre-MDT0000 complete [ 228.939396] LustreError: 16680:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768140211 with bad export cookie 6743050382008246343 [ 228.942658] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 228.944548] LustreError: 16680:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 229.066813] Lustre: server umount lustre-MDT0001 complete [ 230.860133] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:03:33 (1768140213) [ 231.818346] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 232.535675] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:03:34 (1768140214) [ 235.205709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 235.361752] LustreError: 18941:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 235.390510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 236.858438] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 240.279264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 242.175371] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 244.509166] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 245.995651] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 248.674140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 251.036487] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 253.061504] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 270.308552] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 270.308658] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 270.321501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 270.323152] Lustre: Skipped 1 previous similar message [ 270.328516] Lustre: Skipped 6 previous similar messages [ 273.591117] LustreError: 21200:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 273.593760] LustreError: 21200:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 273.651798] Lustre: server umount lustre-OST0000 complete [ 276.448288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 280.560157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 280.565324] Lustre: Skipped 1 previous similar message [ 281.914312] Lustre: server umount lustre-MDT0000 complete [ 283.722987] LustreError: 18922:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768140266 with bad export cookie 6743050382008247225 [ 283.728844] LustreError: 18922:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 283.732715] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 283.905534] Lustre: server umount lustre-MDT0001 complete [ 289.256120] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 290.500059] Key type lgssc unregistered [ 290.634450] LNet: 22135:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.637198] LNetError: 22135:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 290.643355] LNet: Removed LNI 192.168.203.104@tcp [ 290.995111] Key type .llcrypt unregistered [ 290.996232] Key type ._llcrypt unregistered [ 300.921936] Key type ._llcrypt registered [ 300.923181] Key type .llcrypt registered [ 300.964943] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 301.296303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 301.332034] alg: No test for adler32 (adler32-zlib) [ 302.192802] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 302.286252] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 303.903161] Key type lgssc registered [ 304.399379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 308.617303] loop: module loaded [ 308.642673] loop0: detected capacity change from 0 to 204800000 [ 308.767712] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 308.771905] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 308.906573] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 311.533872] loop0: detected capacity change from 0 to 204800000 [ 311.608953] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 311.613589] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 311.622675] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 314.027766] loop0: detected capacity change from 0 to 204800000 [ 316.630189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 316.645350] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 317.760150] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 317.784063] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 317.830615] Lustre: lustre-MDT0000: new disk, initializing [ 317.901927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 317.921184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 319.922546] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 324.483706] loop1: detected capacity change from 0 to 204800000 [ 326.643903] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 326.772846] Lustre: lustre-OST07c6: new disk, initializing [ 326.776108] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 326.778852] Lustre: Skipped 1 previous similar message [ 326.808946] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 328.936278] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 329.224167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 329.228191] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 329.249420] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 332.260593] Lustre: 24265:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 332.263526] Lustre: 24265:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 332.265792] Lustre: 24265:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 332.269228] Lustre: 24265:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 332.272470] Lustre: 24265:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 332.275302] Lustre: 24265:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 349.968456] Lustre: 24264:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 349.972833] Lustre: 24264:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 349.975336] Lustre: 24264:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 349.977341] Lustre: 24264:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 349.981653] Lustre: 24264:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 349.984921] Lustre: 24264:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 353.831708] Lustre: 24265:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 353.834988] Lustre: 24265:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 353.837537] Lustre: 24265:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 353.840178] Lustre: 24265:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 353.842619] Lustre: 24265:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 353.845189] Lustre: 24265:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 353.847646] Lustre: 24265:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 353.850269] Lustre: 24265:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 353.853024] Lustre: 24265:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 353.854958] Lustre: 24265:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 353.857993] Lustre: 24265:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 353.860860] Lustre: 24265:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 356.938707] Lustre: 24265:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 356.943368] Lustre: 24265:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 356.946812] Lustre: 24265:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 356.950433] Lustre: 24265:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 356.954135] Lustre: 24265:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 356.962106] Lustre: 24265:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 356.969417] Lustre: 24265:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 356.974524] Lustre: 24265:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 356.978193] Lustre: 24265:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 356.980461] Lustre: 24265:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 356.982903] Lustre: 24265:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 356.985290] Lustre: 24265:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 363.715864] Lustre: 24264:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 363.719334] Lustre: 24264:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 3 previous similar messages [ 363.722160] Lustre: 24264:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 363.725274] Lustre: 24264:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 363.728332] Lustre: 24264:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 363.730726] Lustre: 24264:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 363.733908] Lustre: 24264:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 363.737728] Lustre: 24264:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 363.740702] Lustre: 24264:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 363.743551] Lustre: 24264:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 363.746868] Lustre: 24264:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 363.749611] Lustre: 24264:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 373.490779] Lustre: 24263:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 373.494748] Lustre: 24263:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 5 previous similar messages [ 373.497840] Lustre: 24263:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 373.501070] Lustre: 24263:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 373.504332] Lustre: 24263:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 373.507546] Lustre: 24263:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 373.510908] Lustre: 24263:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 373.513791] Lustre: 24263:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 373.516258] Lustre: 24263:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 373.520837] Lustre: 24263:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 373.524778] Lustre: 24263:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 373.528642] Lustre: 24263:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 416.225639] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 416.226284] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 416.232644] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 421.106145] LustreError: 26469:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 421.147047] Lustre: server umount lustre-OST07c6 complete [ 425.419511] LustreError: 26970:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 425.422250] LustreError: 26970:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 425.493596] Lustre: server umount lustre-MDT0000 complete [ 431.886655] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 433.010384] Key type lgssc unregistered [ 433.143548] LNet: 27801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 433.146406] LNetError: 27801:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 434.213384] LNet: Removed LNI 192.168.203.104@tcp [ 434.519382] Key type .llcrypt unregistered [ 434.521133] Key type ._llcrypt unregistered [ 442.263398] Key type ._llcrypt registered [ 442.264500] Key type .llcrypt registered [ 442.300475] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 442.725857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 442.735707] alg: No test for adler32 (adler32-zlib) [ 443.589832] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 443.670980] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 445.255159] Key type lgssc registered [ 445.627205] Lustre: Echo OBD driver; http://www.lustre.org/ [ 448.306388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 448.313888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 449.478655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.885866] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 453.911579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 454.029651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 455.436991] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 457.379938] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 458.797752] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 461.187167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 461.301779] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 463.289991] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 465.183839] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 474.599747] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 483.808209] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 483.810889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 483.815949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 484.838791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 484.841811] Lustre: Skipped 2 previous similar messages [ 486.576111] LustreError: 31494:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 486.619047] Lustre: server umount lustre-OST0000 complete [ 489.956649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 489.960135] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 489.960507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 489.961715] Lustre: Skipped 2 previous similar messages [ 489.968196] LustreError: Skipped 2 previous similar messages [ 494.206682] LustreError: 31695:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 494.208605] LustreError: 31695:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 494.237407] Lustre: server umount lustre-MDT0000 complete [ 495.072123] LustreError: 29752:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.077681] LustreError: 29752:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 3 previous similar messages [ 495.564033] LustreError: 29010:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768140478 with bad export cookie 17875683666272915062 [ 495.564970] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 495.567417] LustreError: 29010:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 495.639148] LustreError: 31898:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 495.641414] LustreError: 31898:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 495.687426] Lustre: server umount lustre-MDT0001 complete [ 499.617359] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 500.711087] Key type lgssc unregistered [ 500.841453] LNet: 32428:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 500.845340] LNetError: 32428:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 500.854397] LNet: Removed LNI 192.168.203.104@tcp [ 501.158202] Key type .llcrypt unregistered [ 501.159707] Key type ._llcrypt unregistered [ 509.400920] Key type ._llcrypt registered [ 509.402201] Key type .llcrypt registered [ 509.447530] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 509.823778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 509.867802] alg: No test for adler32 (adler32-zlib) [ 510.745907] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 510.848730] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 512.447169] Key type lgssc registered [ 512.866820] Lustre: Echo OBD driver; http://www.lustre.org/ [ 515.751707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 515.759317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 516.942509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 518.376884] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 521.593459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 521.737462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 523.161505] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 525.204890] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 526.681616] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 529.472192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 529.595727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 531.678731] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 533.642548] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 542.947721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 552.416407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 552.418864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 552.422662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 553.441406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 553.441545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.442315] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 555.503091] LustreError: 35919:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 555.547935] Lustre: server umount lustre-OST0000 complete [ 557.535815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 557.538194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 557.542541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 557.544418] Lustre: Skipped 1 previous similar message [ 563.680273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 563.682776] Lustre: Skipped 3 previous similar messages [ 568.800297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 568.802774] Lustre: Skipped 1 previous similar message [ 571.359111] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 571.395674] LustreError: 36121:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 571.398264] LustreError: 36121:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 571.428750] Lustre: server umount lustre-MDT0000 complete [ 572.747320] LustreError: 33682:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768140555 with bad export cookie 13394320186971080845 [ 572.751299] LustreError: 33682:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 572.751353] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 572.819482] LustreError: 36323:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 572.821736] LustreError: 36323:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 572.876991] Lustre: server umount lustre-MDT0001 complete [ 577.037540] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 578.161595] Key type lgssc unregistered [ 578.309467] LNet: 36854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 578.312297] LNetError: 36854:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 578.319440] LNet: Removed LNI 192.168.203.104@tcp [ 578.651969] Key type .llcrypt unregistered [ 578.653569] Key type ._llcrypt unregistered [ 588.398121] Key type ._llcrypt registered [ 588.399511] Key type .llcrypt registered [ 588.439671] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 588.788634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 588.834480] alg: No test for adler32 (adler32-zlib) [ 589.694581] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 589.791759] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 591.383128] Key type lgssc registered [ 591.819986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 594.884688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 594.894935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 596.083740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 597.695931] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 600.992359] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 601.145033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 602.664494] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 604.852740] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 606.368811] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 608.851935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 608.986302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 610.998901] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 612.910979] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 620.262605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 625.632683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 625.633518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 625.636467] Lustre: Skipped 1 previous similar message [ 628.365607] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 628.369696] Lustre: Skipped 2 previous similar messages [ 630.753583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 630.756134] Lustre: Skipped 2 previous similar messages [ 631.808748] LustreError: 40489:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 631.841664] Lustre: server umount lustre-MDT0000 complete [ 633.329314] LustreError: 38208:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768140616 with bad export cookie 2104167984010485541 [ 633.332186] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 633.334483] LustreError: 38208:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 633.472069] LustreError: 39387:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 633.477878] LustreError: 39387:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 2 previous similar messages [ 633.481590] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.4@tcp (stopping) [ 635.872100] LustreError: 39387:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.872815] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 635.880696] Lustre: Skipped 1 previous similar message [ 638.592095] LustreError: 38223:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 638.592785] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.4@tcp (stopping) [ 638.604304] Lustre: Skipped 1 previous similar message [ 639.488369] LustreError: 40692:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 639.490110] LustreError: 40692:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 639.545909] Lustre: server umount lustre-MDT0001 complete [ 657.580649] LustreError: 40915:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 657.582596] LustreError: 40915:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 657.635752] Lustre: server umount lustre-OST0000 complete [ 663.463526] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 664.516359] Key type lgssc unregistered [ 664.665455] LNet: 41751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 664.668049] LNetError: 41751:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 664.677337] LNet: Removed LNI 192.168.203.104@tcp [ 665.013622] Key type .llcrypt unregistered [ 665.014882] Key type ._llcrypt unregistered [ 674.667257] Key type ._llcrypt registered [ 674.668318] Key type .llcrypt registered [ 674.710606] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 675.108771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 675.118692] alg: No test for adler32 (adler32-zlib) [ 675.998325] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 676.090779] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 677.679184] Key type lgssc registered [ 678.077907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 680.895199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 680.903723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 682.085702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 683.470919] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 686.639457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 686.748180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 688.142552] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 689.992571] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 691.431315] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 693.861083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 693.995270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 696.068491] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 697.999284] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 706.277641] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 711.649238] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 711.649550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 711.650798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 711.654110] Lustre: Skipped 1 previous similar message [ 713.866945] Lustre: lustre-OST0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 713.869946] Lustre: Skipped 1 previous similar message [ 716.768399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 716.770733] Lustre: Skipped 1 previous similar message [ 717.167130] LustreError: 45386:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 717.204694] Lustre: server umount lustre-OST0000 complete [ 730.026263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 730.027022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 730.030260] Lustre: Skipped 1 previous similar message [ 730.034378] Lustre: Skipped 1 previous similar message [ 735.200079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 735.201850] Lustre: Skipped 1 previous similar message [ 736.192807] LustreError: 45763:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 736.194941] LustreError: 45763:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 736.220841] Lustre: server umount lustre-MDT0000 complete [ 737.443074] LustreError: 43105:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768140720 with bad export cookie 2422069809602329372 [ 737.445929] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 737.446543] LustreError: 43105:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 743.548584] LustreError: 45965:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 743.552090] LustreError: 45965:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 743.600643] Lustre: server umount lustre-MDT0001 complete [ 746.996199] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 747.970605] Key type lgssc unregistered [ 748.093368] LNet: 46497:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 748.096629] LNetError: 46497:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 748.104293] LNet: Removed LNI 192.168.203.104@tcp [ 748.387844] Key type .llcrypt unregistered [ 748.389420] Key type ._llcrypt unregistered [ 757.186818] Key type ._llcrypt registered [ 757.188037] Key type .llcrypt registered [ 757.223795] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 757.544903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 757.586054] alg: No test for adler32 (adler32-zlib) [ 758.441704] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 758.523201] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 760.103161] Key type lgssc registered [ 760.470763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 763.079795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 763.085165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 764.264369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 765.617622] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 768.460777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 768.577665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 769.855853] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 771.631890] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 772.863237] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 775.246537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 775.384789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 777.292457] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 779.083995] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 789.733565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 796.727147] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 797.789648] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 808.476412] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-01-11 9h13m10s [ 808.942429] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 810.119083] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 810.465638] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 810.469620] Lustre: Skipped 1 previous similar message [ 811.135237] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 811.137975] Lustre: Skipped 3 previous similar messages [ 815.739216] LustreError: 50451:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 815.764939] Lustre: server umount lustre-MDT0000 complete [ 816.609153] LustreError: 47865:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 816.613596] LustreError: 47865:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 2 previous similar messages [ 816.808926] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 817.279139] LustreError: 49199:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 818.825754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 818.857214] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 818.934357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 818.957930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 819.942326] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 820.169511] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 821.220053] Lustre: DEBUG MARKER: Wait for df (36006) ... [ 821.672670] Lustre: DEBUG MARKER: done [ 824.287978] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 824.289990] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 824.292806] Lustre: Skipped 1 previous similar message [ 824.294263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 824.295774] Lustre: Skipped 3 previous similar messages [ 824.799982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 828.653903] LustreError: 51495:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 828.656719] LustreError: 51495:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 828.696691] Lustre: server umount lustre-OST0000 complete [ 830.433128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 830.433308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 830.433703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 830.433707] Lustre: Skipped 3 previous similar messages [ 830.436858] Lustre: Skipped 2 previous similar messages [ 835.488109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 835.489606] Lustre: Skipped 3 previous similar messages [ 844.255122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 844.285445] LustreError: 51700:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 844.287585] LustreError: 51700:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 844.312912] Lustre: server umount lustre-MDT0000 complete [ 845.545043] LustreError: 47851:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768140828 with bad export cookie 13018196274248666097 [ 845.547696] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 845.548543] LustreError: 47851:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 845.660559] Lustre: server umount lustre-MDT0001 complete [ 849.176861] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 850.127951] Key type lgssc unregistered [ 850.249438] LNet: 52433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.252071] LNetError: 52433:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.262304] LNet: Removed LNI 192.168.203.104@tcp [ 850.536827] Key type .llcrypt unregistered [ 850.537893] Key type ._llcrypt unregistered [ 864.398255] Key type ._llcrypt registered [ 864.399356] Key type .llcrypt registered [ 864.435139] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 864.788338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 864.854365] alg: No test for adler32 (adler32-zlib) [ 865.705437] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 865.788485] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 867.367142] Key type lgssc registered [ 867.701266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 870.169846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 870.175973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 871.268473] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 871.276270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 871.335151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 872.561166] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 875.178229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 875.207019] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 875.216856] Lustre: 54598:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 875.304309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 876.486200] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 878.061278] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 879.104733] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 881.106750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 881.182849] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 881.237903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 882.908816] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 884.560696] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 886.798590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 893.167086] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-01-11 9h14m35s [ 893.672626] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 895.974224] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 896.786433] Lustre: DEBUG MARKER: Creating a test file and stat it [ 896.803307] LustreError: 54603:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 917.131201] Lustre: lustre-MDT0000: Client 5b69fd00-34d2-40a0-9aee-3f30e545fca3 (at 192.168.203.4@tcp) reconnecting [ 936.847099] LustreError: 54603:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 937.543661] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 938.339593] Lustre: DEBUG MARKER: done [ 943.073908] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 943.074753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 943.075397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 943.075956] LustreError: Skipped 1 previous similar message [ 943.079375] Lustre: Skipped 1 previous similar message [ 946.028203] LustreError: 57630:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 946.055469] Lustre: server umount lustre-OST0000 complete [ 947.167729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 947.169985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 947.173839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 947.176342] Lustre: Skipped 1 previous similar message [ 948.192349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 948.194760] Lustre: Skipped 2 previous similar messages [ 953.248264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 953.250020] Lustre: Skipped 1 previous similar message [ 953.276511] LustreError: 57831:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 953.278079] LustreError: 57831:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 953.301649] Lustre: server umount lustre-MDT0000 complete [ 954.458091] LustreError: 55323:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768140937 with bad export cookie 5249816710093489898 [ 954.461499] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 954.461635] LustreError: 55323:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 954.521669] LustreError: 58035:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 954.523430] LustreError: 58035:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 954.571468] Lustre: server umount lustre-MDT0001 complete [ 958.085126] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 958.999970] Key type lgssc unregistered [ 959.117417] LNet: 58565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.119552] LNetError: 58565:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.127289] LNet: Removed LNI 192.168.203.104@tcp [ 959.389235] Key type .llcrypt unregistered [ 959.390440] Key type ._llcrypt unregistered [ 974.200129] Key type ._llcrypt registered [ 974.201055] Key type .llcrypt registered [ 974.237838] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 974.560120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 974.580060] alg: No test for adler32 (adler32-zlib) [ 975.429597] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 975.512449] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 977.095175] Key type lgssc registered [ 977.488216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 978.573927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 978.578880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 979.660639] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 979.667612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 979.720691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 980.021146] Lustre: Failing over lustre-MDT0000 [ 980.076416] LustreError: 60764:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 980.113506] Lustre: server umount lustre-MDT0000 complete [ 982.139235] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:16:04 (1768140964) [ 984.175633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 984.302118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 985.454918] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 987.938306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 987.966918] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 987.978713] Lustre: 61320:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 988.069071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 989.201366] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 990.770786] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 991.887026] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 994.060763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 994.126308] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 994.162938] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 994.214632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 996.117375] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 997.871173] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 999.400345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1005.991910] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1008.607914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1008.610192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1008.614675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1009.633341] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1009.634224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1009.636865] Lustre: Skipped 1 previous similar message [ 1009.639740] Lustre: Skipped 1 previous similar message [ 1013.761717] LustreError: 63693:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 1013.763817] LustreError: 63693:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1013.793476] Lustre: server umount lustre-MDT0000 complete [ 1015.030249] LustreError: 61311:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768140997 with bad export cookie 5187582496625313135 [ 1015.035737] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1015.110895] LustreError: 63894:0:(obd_class.h:479:obd_check_dev()) Device 17 not setup [ 1015.113597] LustreError: 63894:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1015.170709] Lustre: server umount lustre-MDT0001 complete [ 1016.292288] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1017.871689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1020.676798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1024.991928] LustreError: 60244:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fcaac621 x1854030173205504/t0(0) o250->MGC192.168.203.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1025.129578] LustreError: 62935:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1025.130199] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1025.136187] LustreError: 62935:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 4 previous similar messages [ 1025.163172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1025.192407] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1026.439597] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1029.288531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1030.642578] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1032.468546] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1033.725578] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1034.725389] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1035.743251] Lustre: 60245:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768141002/real 1768141002] req@00000000793b5ab2 x1854030173205120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1768141018 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1036.767123] Lustre: 60246:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768141003/real 1768141003] req@00000000ad06cb61 x1854030173205248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1768141019 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1039.845972] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1041.503129] Lustre: 60247:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768141007/real 1768141007] req@00000000365bb98a x1854030173205376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1768141023 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1044.960974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1044.962103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1044.970551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1044.972938] Lustre: Skipped 2 previous similar messages [ 1046.592489] LustreError: 66148:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1046.594509] LustreError: 66148:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1046.620142] Lustre: server umount lustre-MDT0000 complete [ 1047.904337] LustreError: 61313:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768141030 with bad export cookie 5187582496625315368 [ 1047.908117] LustreError: 61313:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1047.909551] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1048.035673] Lustre: server umount lustre-MDT0001 complete [ 1050.195315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1053.397343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1057.183755] LustreError: 60244:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005ebc7693 x1854030173225088/t0(0) o250->MGC192.168.203.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1057.318899] LustreError: 62934:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1057.346585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1057.350183] Lustre: Skipped 1 previous similar message [ 1057.373717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1058.736461] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1061.574465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1061.664236] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1061.670476] Lustre: Skipped 2 previous similar messages [ 1062.985111] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1064.806953] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1064.837130] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1065.439099] Lustre: 60247:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768141032/real 1768141032] req@00000000945bc34c x1854030173224704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1768141048 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1066.025398] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1070.049475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1070.050720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1070.052954] Lustre: Skipped 2 previous similar messages [ 1070.055907] Lustre: Skipped 2 previous similar messages [ 1072.095173] Lustre: 60245:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768141038/real 1768141038] req@00000000ed360e23 x1854030173224960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1768141054 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1072.106501] Lustre: 60245:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1073.855943] LustreError: 68595:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1073.858468] LustreError: 68595:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1073.885535] Lustre: server umount lustre-MDT0000 complete [ 1075.123131] LustreError: 61352:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768141058 with bad export cookie 5187582496625316740 [ 1075.126495] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1075.127147] LustreError: 61352:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1075.167732] LustreError: 67201:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1075.237258] Lustre: server umount lustre-MDT0001 complete [ 1077.411461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1078.515953] Lustre: DEBUG MARKER: files compared the same [ 1081.243115] Lustre: server umount lustre-OST0000 complete [ 1086.417114] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1087.431689] Key type lgssc unregistered [ 1087.557410] LNet: 70224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1087.561363] LNetError: 70224:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1087.570331] LNet: Removed LNI 192.168.203.104@tcp [ 1087.862636] Key type .llcrypt unregistered [ 1087.863769] Key type ._llcrypt unregistered [ 1096.930063] Key type ._llcrypt registered [ 1096.931057] Key type .llcrypt registered [ 1096.970366] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1097.296314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1097.323069] alg: No test for adler32 (adler32-zlib) [ 1098.165657] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 1098.250683] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1099.823178] Key type lgssc registered [ 1100.193810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1102.780074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1102.788958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1103.957867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1105.242792] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug +malloc all [ 1108.078291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1108.175863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1109.527869] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug +malloc all [ 1111.331357] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1112.645760] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1114.916649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1115.008351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1116.881288] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug +malloc all [ 1118.623142] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1128.354720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1133.535749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1133.536529] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1133.538258] LustreError: Skipped 1 previous similar message [ 1133.542097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1133.544749] Lustre: Skipped 1 previous similar message [ 1137.582089] LustreError: 73856:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1137.617105] Lustre: server umount lustre-OST0000 complete [ 1139.167575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1139.170256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1139.173845] Lustre: Skipped 1 previous similar message [ 1139.175744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1143.775558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1143.775719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.956418] LustreError: 74059:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1144.958836] LustreError: 74059:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1144.984813] Lustre: server umount lustre-MDT0000 complete [ 1146.260644] LustreError: 72313:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768141129 with bad export cookie 15963412250473050461 [ 1146.262288] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1146.265952] LustreError: 72313:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1146.333717] LustreError: 74261:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1146.336144] LustreError: 74261:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1146.380456] Lustre: server umount lustre-MDT0001 complete [ 1149.664421] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1150.604947] Key type lgssc unregistered [ 1150.720310] LNet: 74793:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1150.722087] LNetError: 74793:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1150.727242] LNet: Removed LNI 192.168.203.104@tcp [ 1150.983132] Key type .llcrypt unregistered [ 1150.984084] Key type ._llcrypt unregistered [ 1159.245733] Key type ._llcrypt registered [ 1159.246638] Key type .llcrypt registered [ 1159.283195] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1159.579115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1159.642578] alg: No test for adler32 (adler32-zlib) [ 1160.497953] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 1160.576038] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1162.151163] Key type lgssc registered [ 1162.475970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1164.881183] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1164.887041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1180.319290] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1185.439153] LustreError: 76140:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [ 1212.063133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1213.972245] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1215.717948] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1218.232722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1218.279702] Lustre: *** cfs_fail_loc=706, val=0*** [ 1218.421172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1218.451210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1219.732087] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1222.596500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1222.721700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1223.999985] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1225.828135] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1227.077863] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1228.767870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1228.770160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1228.774292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1232.353637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1232.355543] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1237.474755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1237.476738] Lustre: Skipped 2 previous similar messages [ 1242.079095] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1242.099113] LustreError: 78459:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1242.130287] Lustre: server umount lustre-OST0000 complete [ 1247.713024] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1247.713919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1247.717729] Lustre: Skipped 1 previous similar message [ 1249.468394] LustreError: 78664:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1249.470072] LustreError: 78664:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1249.498954] Lustre: server umount lustre-MDT0000 complete [ 1250.736160] LustreError: 77140:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768141233 with bad export cookie 16493185694035772382 [ 1250.740532] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1250.740874] LustreError: 77140:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1250.804988] LustreError: 78866:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1250.807222] LustreError: 78866:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1250.856371] Lustre: server umount lustre-MDT0001 complete [ 1254.418365] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1255.442311] Key type lgssc unregistered [ 1255.563334] LNet: 79397:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1255.566572] LNetError: 79397:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1255.578261] LNet: Removed LNI 192.168.203.104@tcp [ 1255.870948] Key type .llcrypt unregistered [ 1255.872508] Key type ._llcrypt unregistered [ 1264.943081] Key type ._llcrypt registered [ 1264.944058] Key type .llcrypt registered [ 1264.984261] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1265.295098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1265.328820] alg: No test for adler32 (adler32-zlib) [ 1266.175892] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 1266.276072] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1267.863147] Key type lgssc registered [ 1268.250612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1270.911352] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1270.919979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1273.283816] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1275.833398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1276.003902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1277.272433] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1280.062922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1280.168099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1281.954297] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1284.761131] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1284.765854] Lustre: MGS: Client c58409ab-663b-4359-8112-8ffda1f15708 (at 0@lo) reconnecting [ 1284.853285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1284.887777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1286.095986] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1295.328255] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1295.330639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1295.335112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1296.354376] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1296.354825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1296.354976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1299.885129] LustreError: 83400:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1299.916887] Lustre: server umount lustre-OST0000 complete [ 1301.472918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1301.473344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1301.473805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1301.473810] Lustre: Skipped 1 previous similar message [ 1301.481461] Lustre: Skipped 1 previous similar message [ 1306.592246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1306.594476] Lustre: Skipped 2 previous similar messages [ 1307.278315] LustreError: 83602:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1307.283211] LustreError: 83602:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1307.298515] Lustre: server umount lustre-MDT0000 complete [ 1308.607803] LustreError: 83804:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1308.609469] LustreError: 83804:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1308.654197] Lustre: server umount lustre-MDT0001 complete [ 1309.900329] Lustre: server umount MGS complete [ 1313.594420] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:21:35 (1768141295) [ 1322.840895] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 1326.202185] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1337.661142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1338.909255] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1342.663447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1342.766512] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1342.784531] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1342.831718] Lustre: lustre-MDT0001: new disk, initializing [ 1342.859250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1342.870666] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1342.872686] Lustre: 88500:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1342.874736] LustreError: 88500:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1344.160091] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1348.328325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1348.419710] Lustre: lustre-OST0000: new disk, initializing [ 1348.421537] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1348.442773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1350.259738] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1351.392052] Lustre: 89597:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000f3de2c52 x1854030478345088/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1768141350 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1353.416842] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1355.688709] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1355.694998] Lustre: MGS: Client 3002a86f-3f22-488e-8183-3c7ee50ad777 (at 0@lo) reconnecting [ 1355.715220] Lustre: 87787:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1355.725610] LustreError: 87787:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1355.738946] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1355.784677] Lustre: lustre-MDT0000: new disk, initializing [ 1355.838564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1356.895709] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1356.895954] Lustre: 90195:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000e6db6ff4 x1854030478350592/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1768141355 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1356.903343] Lustre: 90195:0:(client.c:1637:after_reply()) Skipped 1 previous similar message [ 1357.061990] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1360.481977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1360.484590] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1361.953544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 1361.955786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1361.957671] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1381.344116] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1381.346223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1381.349822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1386.668074] LustreError: 90727:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1386.671103] LustreError: 90727:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1386.708515] Lustre: server umount lustre-OST0000 complete [ 1390.560582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1390.560959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1390.561580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1390.561584] Lustre: Skipped 3 previous similar messages [ 1390.562854] LustreError: Skipped 1 previous similar message [ 1390.565682] Lustre: Skipped 2 previous similar messages [ 1402.335069] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1402.367227] LustreError: 90929:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1402.369055] LustreError: 90929:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1402.376679] Lustre: server umount lustre-MDT0000 complete [ 1409.762431] Lustre: server umount lustre-MDT0001 complete [ 1410.947987] LustreError: 91332:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1410.949661] LustreError: 91332:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1410.969644] Lustre: server umount MGS complete [ 1414.678072] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:23:16 (1768141396) [ 1420.483787] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1421.574637] Key type lgssc unregistered [ 1421.716376] LNet: 92666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1421.719043] LNetError: 92666:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1421.725311] LNet: Removed LNI 192.168.203.104@tcp [ 1422.023439] Key type .llcrypt unregistered [ 1422.025353] Key type ._llcrypt unregistered [ 1429.197804] Key type ._llcrypt registered [ 1429.199187] Key type .llcrypt registered [ 1429.243361] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1429.690624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1429.711145] alg: No test for adler32 (adler32-zlib) [ 1430.566416] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 1430.651415] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1432.239154] Key type lgssc registered [ 1432.617702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1435.236099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1435.243537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1436.412136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1437.724995] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1440.607346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1440.731095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1442.003207] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1443.843598] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1445.054534] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1447.296519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1447.415898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1449.322162] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1451.116303] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1461.734578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1465.824054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1465.826419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1465.832764] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1466.849240] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1466.850571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1466.850752] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1466.859445] Lustre: Skipped 1 previous similar message [ 1470.446647] LustreError: 96118:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1470.480722] Lustre: server umount lustre-OST0000 complete [ 1471.968397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1471.972514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1476.575937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1476.578618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1476.582724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1477.823739] LustreError: 96319:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1477.825526] LustreError: 96319:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1477.851630] Lustre: server umount lustre-MDT0000 complete [ 1479.028181] LustreError: 93826:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768141461 with bad export cookie 8559481261168829000 [ 1479.031637] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1479.032792] LustreError: 93826:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1479.100744] LustreError: 96521:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1479.102652] LustreError: 96521:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1479.148236] Lustre: server umount lustre-MDT0001 complete [ 1482.440863] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1483.378358] Key type lgssc unregistered [ 1483.496322] LNet: 97052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1483.498697] LNetError: 97052:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1483.507300] LNet: Removed LNI 192.168.203.104@tcp [ 1483.786700] Key type .llcrypt unregistered [ 1483.788369] Key type ._llcrypt unregistered [ 1489.757471] Key type ._llcrypt registered [ 1489.758344] Key type .llcrypt registered [ 1489.794864] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1490.160768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1490.197680] alg: No test for adler32 (adler32-zlib) [ 1491.041107] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 1491.114200] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1492.687123] Key type lgssc registered [ 1493.038095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1495.053559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1495.053619] LustreError: 98093:0:(tgt_mount.c:2439:server_fill_super()) cfs_race id 716 sleeping [ 1495.055598] Lustre: Skipped 1 previous similar message [ 1495.059093] LustreError: 98094:0:(tgt_mount.c:2439:server_fill_super()) cfs_fail_race id 716 waking [ 1495.060883] LustreError: 98093:0:(tgt_mount.c:2439:server_fill_super()) cfs_fail_race id 716 awake: rc=4998 [ 1495.063561] LustreError: 98093:0:(tgt_mount.c:2451:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1495.066718] LustreError: 98093:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1495.066982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1496.235707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1498.820924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1498.930314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1500.230580] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1501.954977] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing lsmod [ 1502.957890] LustreError: 99264:0:(tgt_mount.c:2439:server_fill_super()) cfs_race id 716 sleeping [ 1502.963709] LustreError: 99265:0:(tgt_mount.c:2439:server_fill_super()) cfs_fail_race id 716 waking [ 1502.966484] LustreError: 99264:0:(tgt_mount.c:2439:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 1502.969644] LustreError: 99264:0:(tgt_mount.c:2451:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1502.973023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1502.973296] LustreError: 99264:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1503.102294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1504.223814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1504.226201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1504.229859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1508.321978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1508.324274] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1508.327970] Lustre: Skipped 1 previous similar message [ 1510.141307] LustreError: 99449:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1510.169439] Lustre: server umount lustre-MDT0000 complete [ 1511.371442] LustreError: 98110:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768141494 with bad export cookie 18212364593212225547 [ 1511.376906] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1512.415226] LustreError: 98562:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000024422cde x1854030714255744/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1512.426624] LustreError: 99654:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1512.428284] LustreError: 99654:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1512.473937] Lustre: server umount lustre-MDT0001 complete [ 1523.890290] LustreError: 99855:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1523.891786] LustreError: 99855:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1523.936195] Lustre: server umount lustre-OST0000 complete [ 1526.780648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1526.959304] LustreError: 100331:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1526.964883] LustreError: 100331:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 2 previous similar messages [ 1526.982744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1528.286870] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1531.121752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1531.252290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1532.553473] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1534.366269] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1535.570945] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1537.869444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1539.965225] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1541.713427] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1551.335214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1556.449370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1556.450287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1556.451554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1556.451557] Lustre: Skipped 2 previous similar messages [ 1556.457232] Lustre: Skipped 1 previous similar message [ 1559.340880] LustreError: 102593:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1559.344066] LustreError: 102593:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1559.384302] Lustre: server umount lustre-OST0000 complete [ 1561.570273] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1561.571224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1561.576416] Lustre: Skipped 1 previous similar message [ 1561.579505] Lustre: Skipped 1 previous similar message [ 1566.688060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1566.689774] Lustre: Skipped 2 previous similar messages [ 1566.785582] LustreError: 102795:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1566.787312] LustreError: 102795:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1566.811731] Lustre: server umount lustre-MDT0000 complete [ 1568.042496] LustreError: 100312:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768141550 with bad export cookie 18212364593212226205 [ 1568.046516] LustreError: 100312:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 1568.046601] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1568.173951] Lustre: server umount lustre-MDT0001 complete [ 1571.689014] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1572.744486] Key type lgssc unregistered [ 1572.869313] LNet: 103528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1572.871782] LNetError: 103528:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1572.878408] LNet: Removed LNI 192.168.203.104@tcp [ 1573.182793] Key type .llcrypt unregistered [ 1573.183711] Key type ._llcrypt unregistered [ 1582.214819] Key type ._llcrypt registered [ 1582.215882] Key type .llcrypt registered [ 1582.257568] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1582.658631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1582.666280] alg: No test for adler32 (adler32-zlib) [ 1583.529951] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 1583.634001] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1585.231287] Key type lgssc registered [ 1585.741770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1588.849788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1588.857636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1590.046615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1591.333646] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1594.303116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1594.435769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1595.853365] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1597.636034] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1598.853599] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1601.077991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1601.186719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1603.010434] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1604.816952] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1615.528566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1622.541340] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1623.299627] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1625.056542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1625.060904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1625.067499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1626.082208] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1626.082869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1626.083236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1630.316079] LustreError: 107266:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1630.348958] Lustre: server umount lustre-OST0000 complete [ 1635.295940] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1635.298530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1635.303554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1635.305074] Lustre: Skipped 1 previous similar message [ 1637.696950] LustreError: 107471:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1637.699554] LustreError: 107471:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1637.730365] Lustre: server umount lustre-MDT0000 complete [ 1638.950098] LustreError: 104883:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768141621 with bad export cookie 9128556081610972590 [ 1638.952837] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1638.953428] LustreError: 104883:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1639.016639] LustreError: 107674:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1639.018238] LustreError: 107674:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1639.065249] Lustre: server umount lustre-MDT0001 complete [ 1642.668886] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1643.653591] Key type lgssc unregistered [ 1643.777419] LNet: 108205:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1643.781143] LNetError: 108205:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1643.789355] LNet: Removed LNI 192.168.203.104@tcp [ 1644.068182] Key type .llcrypt unregistered [ 1644.069219] Key type ._llcrypt unregistered [ 1650.762942] Key type ._llcrypt registered [ 1650.763885] Key type .llcrypt registered [ 1650.807832] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1651.221072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1651.247348] alg: No test for adler32 (adler32-zlib) [ 1652.114201] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 1652.199233] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1653.783251] Key type lgssc registered [ 1654.169681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1656.794726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1656.803885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1657.950743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1659.244774] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1662.054802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1662.180699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1663.482658] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1665.248679] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1666.422954] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1668.649891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1668.757382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1670.643607] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1672.424328] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1683.110830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1693.665929] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1693.666793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1693.669883] Lustre: Skipped 1 previous similar message [ 1693.670312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1693.672605] Lustre: Skipped 1 previous similar message [ 1696.814092] LustreError: 111659:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1696.849473] Lustre: server umount lustre-OST0000 complete [ 1698.271903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1698.275022] LustreError: Skipped 1 previous similar message [ 1698.276830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1698.282639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1703.904261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1703.906047] Lustre: Skipped 2 previous similar messages [ 1704.125456] LustreError: 111861:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1704.127638] LustreError: 111861:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1704.157456] Lustre: server umount lustre-MDT0000 complete [ 1705.355186] LustreError: 109378:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768141688 with bad export cookie 7677634954715621927 [ 1705.357982] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1705.358882] LustreError: 109378:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1705.427843] LustreError: 112062:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1705.429558] LustreError: 112062:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1705.480635] Lustre: server umount lustre-MDT0001 complete [ 1709.112886] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1710.035285] Key type lgssc unregistered [ 1710.156317] LNet: 112593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.158428] LNetError: 112593:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1710.166284] LNet: Removed LNI 192.168.203.104@tcp [ 1710.443045] Key type .llcrypt unregistered [ 1710.444029] Key type ._llcrypt unregistered [ 1719.079548] Key type ._llcrypt registered [ 1719.080475] Key type .llcrypt registered [ 1719.118253] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1719.400536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1719.413685] alg: No test for adler32 (adler32-zlib) [ 1720.258317] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 1720.333482] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1721.911136] Key type lgssc registered [ 1722.233496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1724.611663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1724.617198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1725.761164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1727.053904] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1729.798338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1729.922486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1731.258772] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1733.064883] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1734.268860] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1736.484390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1736.602051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1738.390692] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1740.119551] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1751.974329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 1758.189840] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1759.335983] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1759.339030] Lustre: Skipped 2 previous similar messages [ 1760.674513] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1760.677754] Lustre: Skipped 2 previous similar messages [ 1766.392334] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1766.401289] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1769.401401] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.4@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1769.406158] Lustre: Skipped 2 previous similar messages [ 1775.095196] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.4@tcp 192.168.0.[2,10]@tcp [ 1777.232127] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.4/24@tcp in log lustre-MDT0000 [ 1777.234809] Lustre: Skipped 2 previous similar messages [ 1783.799972] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.4/24@tcp [ 1783.802411] Lustre: Skipped 1 previous similar message [ 1787.876045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1787.880272] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1787.880362] Lustre: Skipped 1 previous similar message [ 1787.881727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1792.176141] LustreError: 117971:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1792.210602] Lustre: server umount lustre-OST0000 complete [ 1796.576563] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1796.580702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1796.587826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1796.592039] Lustre: Skipped 1 previous similar message [ 1798.112168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1798.112781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1799.614966] LustreError: 118172:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1799.616628] LustreError: 118172:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1799.651544] Lustre: server umount lustre-MDT0000 complete [ 1801.036103] LustreError: 115579:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768141783 with bad export cookie 11064458450072328879 [ 1801.042905] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1801.043591] LustreError: 115579:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1801.134102] LustreError: 118373:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1801.136834] LustreError: 118373:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1801.197928] Lustre: server umount lustre-MDT0001 complete [ 1805.093519] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1806.152457] Key type lgssc unregistered [ 1806.286358] LNet: 118904:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1806.288652] LNetError: 118904:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1806.296292] LNet: Removed LNI 192.168.203.104@tcp [ 1806.620482] Key type .llcrypt unregistered [ 1806.621581] Key type ._llcrypt unregistered [ 1817.472758] Key type ._llcrypt registered [ 1817.473775] Key type .llcrypt registered [ 1817.513856] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1818.046574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1818.060622] alg: No test for adler32 (adler32-zlib) [ 1818.916910] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 1819.015185] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1820.599161] Key type lgssc registered [ 1821.002420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1824.119769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1824.128853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1825.322734] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1825.325130] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1825.331328] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.4/24@tcp [ 1826.664438] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1829.584665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1829.698853] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1829.707257] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.4/24@tcp [ 1829.727869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1831.014867] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1832.948375] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1834.244636] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1836.461227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1836.594304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1838.795168] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1840.615420] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1850.918787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 1854.944329] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1854.948369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1854.955436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1855.971622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1855.971955] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1855.972266] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1855.974422] Lustre: Skipped 1 previous similar message [ 1860.014093] LustreError: 122739:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1860.050960] Lustre: server umount lustre-OST0000 complete [ 1865.696105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1865.699588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1865.704110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1867.456305] LustreError: 122940:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1867.459197] LustreError: 122940:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1867.488444] Lustre: server umount lustre-MDT0000 complete [ 1868.801284] LustreError: 122042:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768141851 with bad export cookie 16861869578522908489 [ 1868.804521] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1868.804770] LustreError: 122042:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1868.876371] LustreError: 123141:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1868.879258] LustreError: 123141:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1868.930668] Lustre: server umount lustre-MDT0001 complete [ 1872.636466] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 1873.741846] Key type lgssc unregistered [ 1873.870409] LNet: 123672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1873.872634] LNetError: 123672:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1873.879307] LNet: Removed LNI 192.168.203.104@tcp [ 1874.172762] Key type .llcrypt unregistered [ 1874.173755] Key type ._llcrypt unregistered [ 1882.775715] Key type ._llcrypt registered [ 1882.776672] Key type .llcrypt registered [ 1882.814025] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 1888.744264] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 1889.088488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1889.126095] alg: No test for adler32 (adler32-zlib) [ 1889.974230] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 1890.056058] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 1891.639213] Key type lgssc registered [ 1892.183238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1905.680095] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1905.686874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1906.810320] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1906.822413] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1906.868923] Lustre: lustre-MDT0000: new disk, initializing [ 1906.898537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1906.906277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1908.300366] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1913.007326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1913.073169] Lustre: 127015:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1913.117425] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1913.119982] Lustre: Skipped 1 previous similar message [ 1913.173379] Lustre: lustre-MDT0001: new disk, initializing [ 1913.217444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1913.234429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1913.240441] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1914.831527] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1918.371396] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1919.883814] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1922.801091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1922.953199] Lustre: lustre-OST0000: new disk, initializing [ 1922.956342] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1922.998860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1924.624359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1924.630254] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1924.695039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1925.613882] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1929.161405] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1930.765615] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1930.827852] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1932.527425] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1932.636162] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1934.820849] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1934.823189] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1934.825809] LustreError: Skipped 1 previous similar message [ 1934.834599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1934.840164] Lustre: Skipped 1 previous similar message [ 1939.567201] LustreError: 129947:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1939.609892] Lustre: server umount lustre-OST0000 complete [ 1944.032742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1944.037768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1944.045253] Lustre: Skipped 1 previous similar message [ 1944.048882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1945.056193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1945.056788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1950.177232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1950.181346] Lustre: Skipped 2 previous similar messages [ 1954.273704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1954.277232] Lustre: Skipped 1 previous similar message [ 1955.295165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1955.350455] LustreError: 130148:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1955.352256] LustreError: 130148:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1955.388744] Lustre: server umount lustre-MDT0000 complete [ 1957.173668] LustreError: 127006:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768141940 with bad export cookie 17679105669052504005 [ 1957.179929] LustreError: 127006:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1957.180245] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1957.283969] LustreError: 130351:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1957.287674] LustreError: 130351:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1957.357587] Lustre: server umount lustre-MDT0001 complete [ 1960.943925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1961.192693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1963.005915] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1966.254661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1966.385666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1967.951370] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1969.850013] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1971.099725] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1973.684089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1976.463302] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1978.555729] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1980.074964] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1986.389708] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 1987.859422] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1987.934227] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1991.437828] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1991.515504] Lustre: lustre-OST0001: new disk, initializing [ 1991.518653] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1991.557981] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1991.560978] Lustre: Skipped 1 previous similar message [ 1993.798224] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 1997.733164] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1998.372918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1998.382161] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1998.437879] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1998.902887] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2000.792037] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2000.862798] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2002.176380] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d6a04e38800.ost_server_uuid 50 [ 2002.699608] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9d6a04e38800.ost_server_uuid in FULL state after 0 sec [ 2008.551072] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2008.551396] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2008.555212] LustreError: Skipped 1 previous similar message [ 2008.563494] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2008.568223] Lustre: Skipped 1 previous similar message [ 2010.997194] LustreError: 135116:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2011.001472] LustreError: 135116:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2011.035977] Lustre: server umount lustre-OST0001 complete [ 2013.152579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2013.157735] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2013.165934] Lustre: Skipped 1 previous similar message [ 2013.669354] LustreError: 132420:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2013.676377] LustreError: 132420:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 3 previous similar messages [ 2018.787671] LustreError: 132420:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2018.793864] LustreError: 132420:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 1 previous similar message [ 2018.802073] LustreError: 135317:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2018.803911] LustreError: 135317:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2018.836075] Lustre: server umount lustre-OST0000 complete [ 2022.879862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2022.883083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2022.888091] Lustre: Skipped 1 previous similar message [ 2026.202173] Lustre: server umount lustre-MDT0000 complete [ 2027.563516] LustreError: 130806:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768142010 with bad export cookie 17679105669052504880 [ 2027.567386] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2027.569708] LustreError: 130806:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2027.656420] LustreError: 135719:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2027.659348] LustreError: 135719:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2027.712863] Lustre: server umount lustre-MDT0001 complete [ 2035.121721] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2036.391563] Key type lgssc unregistered [ 2036.542509] LNet: 136708:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2036.545276] LNetError: 136708:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2036.555463] LNet: Removed LNI 192.168.203.104@tcp [ 2036.920265] Key type .llcrypt unregistered [ 2036.921836] Key type ._llcrypt unregistered [ 2050.814523] Key type ._llcrypt registered [ 2050.815738] Key type .llcrypt registered [ 2050.853933] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 2056.551160] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2056.879794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2056.910080] alg: No test for adler32 (adler32-zlib) [ 2057.749268] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 2057.823860] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2059.399125] Key type lgssc registered [ 2059.760784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2070.970174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2070.980622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2072.097409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2072.112558] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2072.157621] Lustre: lustre-MDT0000: new disk, initializing [ 2072.208058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2072.214826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2073.646337] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2077.972609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2078.019148] Lustre: 140796:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2078.051113] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2078.053584] Lustre: Skipped 1 previous similar message [ 2078.098459] Lustre: lustre-MDT0001: new disk, initializing [ 2078.135579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2078.149940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2078.153284] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2079.454368] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2082.582358] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2083.824070] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2086.315297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2086.426355] Lustre: lustre-OST0000: new disk, initializing [ 2086.428367] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2086.460789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2087.736086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2087.739579] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2087.775776] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2088.255430] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2091.871564] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2093.949301] Lustre: Failing over lustre-OST0000 [ 2093.996288] LustreError: 143228:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2094.041783] Lustre: server umount lustre-OST0000 complete [ 2107.088922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2107.201109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2107.205102] Lustre: Skipped 1 previous similar message [ 2107.213818] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2107.221716] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2108.551592] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2109.143199] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2110.462057] Lustre: Failing over lustre-MDT0000 [ 2110.535416] LustreError: 144198:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2110.537117] LustreError: 144198:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2110.568310] Lustre: server umount lustre-MDT0000 complete [ 2112.480545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2112.484582] Lustre: Skipped 2 previous similar messages [ 2114.527177] Lustre: 139024:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768142081/real 1768142081] req@0000000037818902 x1854031307763072/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1768142097 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2114.527177] Lustre: 139026:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768142081/real 1768142081] req@00000000cf59105f x1854031307762816/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1768142097 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2117.600494] LustreError: 140807:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2117.605064] LustreError: 140807:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 7 previous similar messages [ 2118.783777] LustreError: 141584:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.647224] Lustre: 139026:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1768142086/real 1768142086] req@0000000021dda043 x1854031307764352/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1768142102 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2122.720435] LustreError: 140807:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2122.724624] LustreError: 140807:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 2 previous similar messages [ 2123.555757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2123.596618] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2123.601773] Lustre: 140799:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.203.104@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2123.686708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2123.710843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2123.745605] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2123.905430] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2125.205146] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2128.870685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2128.877297] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2128.879817] Lustre: Skipped 1 previous similar message [ 2128.899647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2133.984551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2133.986837] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2133.988553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2133.990461] Lustre: Skipped 1 previous similar message [ 2135.979117] LustreError: 145079:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2135.981891] LustreError: 145079:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2136.020667] Lustre: server umount lustre-OST0000 complete [ 2139.104707] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2139.105434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2139.106056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2139.106061] Lustre: Skipped 1 previous similar message [ 2139.107195] LustreError: Skipped 1 previous similar message [ 2139.110852] Lustre: Skipped 1 previous similar message [ 2143.486576] LustreError: 145281:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2143.489277] LustreError: 145281:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2143.520904] Lustre: server umount lustre-MDT0000 complete [ 2144.224067] LustreError: 141579:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2144.228630] LustreError: 141579:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 1 previous similar message [ 2144.749263] LustreError: 140788:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768142127 with bad export cookie 6167899674520125023 [ 2144.752117] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2144.753216] LustreError: 140788:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2144.871544] Lustre: server umount lustre-MDT0001 complete [ 2148.466889] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2149.691154] Key type lgssc unregistered [ 2149.879299] LNet: 146013:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2149.881290] LNetError: 146013:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2149.889264] LNet: Removed LNI 192.168.203.104@tcp [ 2150.158749] Key type .llcrypt unregistered [ 2150.160090] Key type ._llcrypt unregistered [ 2169.668296] Key type ._llcrypt registered [ 2169.670093] Key type .llcrypt registered [ 2169.706434] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2170.074314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2170.129305] alg: No test for adler32 (adler32-zlib) [ 2170.994134] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 2171.087219] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2172.679102] Key type lgssc registered [ 2173.034895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2175.652860] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2175.659665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2176.829305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2178.076017] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2181.404991] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2183.686083] LustreError: 147538:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2183.692140] LustreError: 147538:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2183.697146] LustreError: 147538:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2183.792824] LustreError: 148060:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2183.857611] Lustre: server umount lustre-MDT0000 complete [ 2188.758503] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 2191.494215] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2202.648790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2202.742898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2202.753807] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2202.794553] Lustre: lustre-MDT0000: new disk, initializing [ 2202.819249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2202.825950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2204.106076] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2208.217771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2208.254927] Lustre: 150687:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2208.278857] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2208.280569] Lustre: Skipped 1 previous similar message [ 2208.320330] Lustre: lustre-MDT0001: new disk, initializing [ 2208.351273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2208.364660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2208.368154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2209.684661] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2212.684530] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2213.875925] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2216.051475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2216.159566] Lustre: lustre-OST0000: new disk, initializing [ 2216.161543] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2216.196574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2218.022990] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2219.446890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2219.449682] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2219.482369] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2221.009210] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2222.771743] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2223.275542] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:36:45 (1768142205) [ 2285.023576] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2285.025347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2285.028137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2286.047925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2286.048064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2286.049676] Lustre: Skipped 1 previous similar message [ 2286.052949] Lustre: Skipped 1 previous similar message [ 2288.381035] LustreError: 154182:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2288.382977] LustreError: 154182:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2288.431327] Lustre: server umount lustre-MDT0000 complete [ 2289.691735] LustreError: 150680:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768142272 with bad export cookie 9350849745844927888 [ 2289.693049] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2289.694979] LustreError: 150680:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2291.167458] LustreError: 150722:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2291.167514] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2291.173437] LustreError: 150722:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 2 previous similar messages [ 2291.177697] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2294.175879] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2294.175879] LustreError: 152438:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2295.810198] LustreError: 154383:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2295.813150] LustreError: 154383:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2295.862111] Lustre: server umount lustre-MDT0001 complete [ 2303.401873] LustreError: 154584:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2303.404066] LustreError: 154584:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2303.437764] Lustre: server umount lustre-OST0000 complete [ 2306.266388] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 2309.118329] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2321.905586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2321.995740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2322.005941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2322.042956] Lustre: lustre-MDT0000: new disk, initializing [ 2322.065711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2322.071599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2323.351882] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2327.494138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2327.532782] Lustre: 156909:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2327.553509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2327.555245] Lustre: Skipped 1 previous similar message [ 2327.591184] Lustre: lustre-MDT0001: new disk, initializing [ 2327.631729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2327.636063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2329.021654] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2332.492598] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2333.975520] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2336.042606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2336.134112] Lustre: lustre-OST0000: new disk, initializing [ 2336.135934] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2336.162541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2336.164421] Lustre: Skipped 1 previous similar message [ 2337.277352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2337.281242] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2337.328467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2338.015349] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2341.083138] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2342.380311] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2342.457258] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2343.708244] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2343.783985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2347.492821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2347.493333] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2347.495144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2347.495260] LustreError: Skipped 1 previous similar message [ 2347.498977] Lustre: Skipped 1 previous similar message [ 2350.508270] LustreError: 159835:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2350.511272] LustreError: 159835:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2350.548481] Lustre: server umount lustre-OST0000 complete [ 2352.608710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2352.612421] Lustre: Skipped 1 previous similar message [ 2357.728516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2357.730788] Lustre: Skipped 4 previous similar messages [ 2357.935466] Lustre: server umount lustre-MDT0000 complete [ 2359.487279] LustreError: 156899:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768142342 with bad export cookie 9350849745845086704 [ 2359.490368] LustreError: 156899:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2359.491613] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2359.555528] LustreError: 160239:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2359.557618] LustreError: 160239:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2359.610522] Lustre: server umount lustre-MDT0001 complete [ 2362.648747] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:39:04 (1768142344) [ 2368.479927] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 2372.035907] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2384.967982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2385.096935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2385.112700] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2385.178393] Lustre: lustre-MDT0000: new disk, initializing [ 2385.219343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2385.232075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2386.686996] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2391.612068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2391.670522] Lustre: 163211:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2391.790738] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2393.108995] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2396.427356] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2397.617407] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2400.086475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2400.355465] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2400.407976] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2402.124121] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2405.363678] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2410.977748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2410.979257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2410.980486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2410.981172] Lustre: Skipped 1 previous similar message [ 2410.987880] LustreError: Skipped 1 previous similar message [ 2413.613098] LustreError: 165744:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2413.615214] LustreError: 165744:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2413.646285] Lustre: server umount lustre-OST0000 complete [ 2421.038473] Lustre: server umount lustre-MDT0000 complete [ 2421.216851] LustreError: 163222:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2421.226292] LustreError: 163222:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 1 previous similar message [ 2422.767190] LustreError: 163203:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768142405 with bad export cookie 9350849745845087572 [ 2422.770332] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2422.773491] LustreError: 163203:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2422.898855] Lustre: server umount lustre-MDT0001 complete [ 2426.505945] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:40:08 (1768142408) [ 2432.611879] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 2436.245800] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2451.245745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2451.343455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2451.354516] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2451.356120] Lustre: Skipped 3 previous similar messages [ 2451.396465] Lustre: lustre-MDT0000: new disk, initializing [ 2451.397750] Lustre: Skipped 2 previous similar messages [ 2451.421248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2451.424517] Lustre: Skipped 2 previous similar messages [ 2451.430759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2451.433186] Lustre: Skipped 2 previous similar messages [ 2452.764709] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2457.031532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2457.072606] Lustre: 169123:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2457.167392] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2458.562169] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2461.698170] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2463.228776] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2465.713898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2467.754446] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2467.934111] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2471.066300] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2478.053346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2478.054097] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2478.059432] LustreError: Skipped 1 previous similar message [ 2478.065627] Lustre: Skipped 2 previous similar messages [ 2478.066625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2478.075210] Lustre: Skipped 4 previous similar messages [ 2479.150844] LustreError: 171658:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2479.153689] LustreError: 171658:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2479.194393] Lustre: server umount lustre-OST0000 complete [ 2482.655993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2486.823436] Lustre: server umount lustre-MDT0000 complete [ 2488.013432] LustreError: 169113:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768142470 with bad export cookie 9350849745845088587 [ 2488.018599] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2488.137044] Lustre: server umount lustre-MDT0001 complete [ 2492.127100] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2493.155841] Key type lgssc unregistered [ 2493.281522] LNet: 172592:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2493.283657] LNetError: 172592:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2493.290446] LNet: Removed LNI 192.168.203.104@tcp [ 2493.584350] Key type .llcrypt unregistered [ 2493.585316] Key type ._llcrypt unregistered [ 2501.961441] Key type ._llcrypt registered [ 2501.962437] Key type .llcrypt registered [ 2502.002491] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2502.378332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2502.393071] alg: No test for adler32 (adler32-zlib) [ 2503.253961] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 2503.338918] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2504.919235] Key type lgssc registered [ 2505.248391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2507.687716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2507.694274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2508.849399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2510.077606] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2512.868260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2512.981859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2514.164796] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2515.809754] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2516.900701] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2519.048293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2519.156264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2521.001599] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2522.695716] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2533.477704] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2553.311812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2553.313734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2553.317176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2554.337678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2554.338409] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2554.338784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2555.565409] LustreError: 176234:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2555.594695] Lustre: server umount lustre-OST0000 complete [ 2558.943782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2558.945699] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2558.949090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2558.950662] Lustre: Skipped 1 previous similar message [ 2564.576101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2564.578009] Lustre: Skipped 3 previous similar messages [ 2569.695986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2569.698121] Lustre: Skipped 1 previous similar message [ 2571.231099] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2571.261987] LustreError: 176436:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2571.263755] LustreError: 176436:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2571.285464] Lustre: server umount lustre-MDT0000 complete [ 2572.391038] LustreError: 173948:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768142555 with bad export cookie 18083476240991084640 [ 2572.393514] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2572.394033] LustreError: 173948:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2572.458658] LustreError: 176639:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2572.460955] LustreError: 176639:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2572.508036] Lustre: server umount lustre-MDT0001 complete [ 2575.547979] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 2576.524866] Key type lgssc unregistered [ 2576.648419] LNet: 177170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2576.650833] LNetError: 177170:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2576.661483] LNet: Removed LNI 192.168.203.104@tcp [ 2576.963514] Key type .llcrypt unregistered [ 2576.964507] Key type ._llcrypt unregistered [ 2585.235382] Key type ._llcrypt registered [ 2585.236356] Key type .llcrypt registered [ 2585.268109] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2585.707824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2585.715134] alg: No test for adler32 (adler32-zlib) [ 2586.560218] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 2586.640205] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 2588.215124] Key type lgssc registered [ 2588.592361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2591.022903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2591.028736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2592.179512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2593.318154] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2596.031015] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2596.151597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2597.314699] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2598.944178] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2600.089118] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2602.272547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2602.401631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2603.433067] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2604.228478] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2605.805269] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2611.684689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2611.688159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2611.688202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2611.690055] Lustre: Skipped 1 previous similar message [ 2612.358756] Lustre: lustre-OST0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 2612.360466] Lustre: Skipped 1 previous similar message [ 2614.957115] LustreError: 180782:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2614.999252] Lustre: server umount lustre-OST0000 complete [ 2616.712682] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9d6a06afe800.ost_server_uuid 50 [ 2617.167675] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff9d6a06afe800.ost_server_uuid in DISCONN state after 0 sec [ 2617.662768] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2631.713744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2631.714374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2631.717721] Lustre: Skipped 1 previous similar message [ 2633.722921] LustreError: 181178:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2633.724516] LustreError: 181178:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2633.744941] Lustre: server umount lustre-MDT0000 complete [ 2634.914131] LustreError: 178522:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768142617 with bad export cookie 105236926843130546 [ 2634.916889] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2634.916992] LustreError: 178522:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2634.986109] LustreError: 181380:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2634.988074] LustreError: 181380:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2635.037649] Lustre: server umount lustre-MDT0001 complete [ 2637.690374] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 09:43:40 (1768142620) [ 2639.752310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2639.891052] LustreError: 182050:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2639.895586] LustreError: 182050:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 2 previous similar messages [ 2639.913832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2641.127946] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2643.749786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2645.050810] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2646.686690] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2647.788678] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2649.833787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2649.938054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2649.939962] Lustre: Skipped 1 previous similar message [ 2650.981845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2651.556797] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2653.140966] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2655.245157] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2655.304231] Lustre: lustre-OST0001: new disk, initializing [ 2655.305832] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2655.934167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2655.936901] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2655.970923] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2657.003888] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2659.987124] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2661.348960] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2661.348992] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2661.352904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2661.355977] Lustre: Skipped 2 previous similar messages [ 2665.606861] Lustre: lustre-OST0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 2675.167085] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2675.182109] LustreError: 185244:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2675.184933] LustreError: 185244:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2675.204264] Lustre: server umount lustre-OST0000 complete [ 2675.841372] LustreError: 184682:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2676.921633] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2677.008443] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2678.615222] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2678.740769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2680.961214] LustreError: 183640:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2680.971112] LustreError: 183640:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 2 previous similar messages [ 2686.080592] LustreError: 183641:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.085192] LustreError: 183641:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 2 previous similar messages [ 2692.066358] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2692.066588] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2692.068316] LustreError: Skipped 1 previous similar message [ 2692.068894] LustreError: 184682:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2692.068905] LustreError: 184682:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 2 previous similar messages [ 2692.071310] Lustre: Skipped 1 previous similar message [ 2692.073353] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2692.083456] Lustre: Skipped 6 previous similar messages [ 2694.767136] LustreError: 185949:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2694.770358] LustreError: 185949:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2694.815390] Lustre: server umount lustre-OST0001 complete [ 2697.186138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2697.189080] Lustre: Skipped 1 previous similar message [ 2702.165894] Lustre: server umount lustre-MDT0000 complete [ 2702.304252] LustreError: 182049:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2702.309712] LustreError: 182049:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 2 previous similar messages [ 2703.339340] LustreError: 182029:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768142686 with bad export cookie 105236926843131603 [ 2703.342487] LustreError: 182029:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2703.342989] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2703.410071] LustreError: 186352:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2703.413509] LustreError: 186352:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2703.466756] Lustre: server umount lustre-MDT0001 complete [ 2710.578081] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 09:44:52 (1768142692) [ 2712.600224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2712.688955] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2712.699846] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2712.770942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2712.772731] Lustre: Skipped 1 previous similar message [ 2713.952246] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2716.672730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2716.703831] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2716.716590] Lustre: 187818:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2718.066463] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2719.647943] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2720.726742] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2722.766855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2722.837749] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2724.803973] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2726.696017] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2726.939281] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 2729.091853] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2729.116383] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2729.189890] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2729.192311] Lustre: Skipped 2 previous similar messages [ 2730.987363] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2732.685628] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2737.120074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2737.122277] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2737.126213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2737.127799] Lustre: Skipped 3 previous similar messages [ 2739.630087] LustreError: 191003:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2739.632055] LustreError: 191003:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2739.649640] Lustre: server umount lustre-OST0000 complete [ 2743.266821] LustreError: 189439:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2743.271409] LustreError: 189439:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 1 previous similar message [ 2752.992636] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2752.996892] LustreError: Skipped 1 previous similar message [ 2753.509035] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2753.514710] Lustre: Skipped 2 previous similar messages [ 2756.366777] Lustre: server umount lustre-OST0001 complete [ 2763.835949] Lustre: server umount lustre-MDT0000 complete [ 2765.374876] LustreError: 187809:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768142748 with bad export cookie 105236926843132968 [ 2765.380416] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2765.383572] LustreError: 187809:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2765.534719] Lustre: server umount lustre-MDT0001 complete [ 2774.236544] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 09:45:56 (1768142756) [ 2780.461244] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 2783.561465] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 2795.772583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2795.869258] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2795.882680] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2795.925279] Lustre: lustre-MDT0000: new disk, initializing [ 2795.951114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2795.958450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2797.244262] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2801.559518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2801.604917] Lustre: 195393:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2801.642414] LustreError: 195405:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2801.646484] LustreError: 195405:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 6 previous similar messages [ 2801.646744] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2801.650852] Lustre: Skipped 1 previous similar message [ 2801.688689] Lustre: lustre-MDT0001: new disk, initializing [ 2801.734542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2801.737913] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2803.108741] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2806.382768] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2807.584433] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2809.973575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2810.083451] Lustre: lustre-OST0000: new disk, initializing [ 2810.085382] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2812.063224] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2812.868589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2812.875578] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2812.911783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2815.133395] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2816.440388] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2816.503930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2817.648133] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2817.714871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2820.334073] LustreError: 198326:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2820.335899] LustreError: 198326:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 2820.366236] Lustre: server umount lustre-OST0000 complete [ 2822.112137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2822.114518] LustreError: Skipped 2 previous similar messages [ 2822.116071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2822.119053] Lustre: Skipped 2 previous similar messages [ 2822.120659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2822.123905] Lustre: Skipped 10 previous similar messages [ 2831.354562] Lustre: server umount lustre-MDT0000 complete [ 2832.635456] LustreError: 196170:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768142815 with bad export cookie 105236926843134515 [ 2832.639430] LustreError: 196170:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2832.640213] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2843.150829] Lustre: server umount lustre-MDT0001 complete [ 2846.273955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2847.654598] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2851.011979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2852.441065] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2854.373399] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2855.563681] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2857.776144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2859.793156] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2861.820019] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2863.298819] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2871.591834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2873.046671] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2873.168316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2876.385498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2880.082590] Lustre: server umount lustre-OST0000 complete [ 2881.914371] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2883.238410] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2906.710045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2906.822221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2906.824275] Lustre: Skipped 5 previous similar messages [ 2908.523994] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2910.206183] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2910.756133] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2916.835929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2916.837668] Lustre: Skipped 6 previous similar messages [ 2917.743084] LustreError: 203629:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2917.745945] LustreError: 203629:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 2917.781521] Lustre: server umount lustre-OST0000 complete [ 2921.953828] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2921.957647] Lustre: Skipped 4 previous similar messages [ 2925.142927] Lustre: server umount lustre-MDT0000 complete [ 2926.391455] LustreError: 202003:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768142909 with bad export cookie 105236926843135397 [ 2926.396126] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2926.396388] LustreError: 202003:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2926.527392] Lustre: server umount lustre-MDT0001 complete [ 2929.345697] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 09:48:31 (1768142911) [ 2931.441657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2931.597537] LustreError: 204706:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2931.604655] LustreError: 204706:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 1 previous similar message [ 2932.894884] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2935.600161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2936.907097] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2938.604080] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2939.800464] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2941.997085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2943.867301] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2945.497407] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2946.785298] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2957.074317] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 2958.594192] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2958.680968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2961.675521] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2961.756156] Lustre: lustre-OST0001: new disk, initializing [ 2961.759086] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2963.882155] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 2967.732491] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2969.315383] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2971.664737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2971.670056] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2971.702945] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2972.521931] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2973.908227] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2973.972963] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2976.740371] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2976.742414] LustreError: Skipped 1 previous similar message [ 2980.666781] Lustre: server umount lustre-OST0001 complete [ 2982.533071] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2982.607707] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2983.862315] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2983.938923] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3007.408688] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3009.155564] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3010.785116] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3011.273771] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3017.971746] Lustre: server umount lustre-OST0001 complete [ 3044.319165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3044.368201] Lustre: server umount lustre-OST0000 complete [ 3048.417112] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3048.417743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3048.419162] LustreError: Skipped 1 previous similar message [ 3048.421708] Lustre: Skipped 20 previous similar messages [ 3051.772415] LustreError: 211065:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3051.774235] LustreError: 211065:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 3051.801755] Lustre: server umount lustre-MDT0000 complete [ 3052.978083] LustreError: 204688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768143035 with bad export cookie 105236926843136328 [ 3052.980961] LustreError: 204688:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3052.981733] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3067.359191] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3067.447963] Lustre: server umount lustre-MDT0001 complete [ 3074.557743] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 09:50:56 (1768143056) [ 3076.578047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3076.663649] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3076.672843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3076.736870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3076.738688] Lustre: Skipped 5 previous similar messages [ 3077.903103] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3080.473979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3080.504962] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3080.516142] Lustre: 212737:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3082.123892] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3083.974983] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3085.621946] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3088.353772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3088.444294] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3091.237117] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3093.723444] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3098.917494] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3098.954964] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3101.721759] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3104.300833] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3105.102113] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3106.710374] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3107.805689] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3109.642422] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3109.753842] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3111.501390] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d6a05a7d000.ost_server_uuid 50 [ 3112.156927] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9d6a05a7d000.ost_server_uuid in FULL state after 0 sec [ 3112.589082] Lustre: Permanently deactivating lustre-OST0001 [ 3112.600233] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3113.426157] Lustre: Permanently reactivating lustre-OST0001 [ 3117.032505] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3117.038064] Lustre: Skipped 7 previous similar messages [ 3120.772161] Lustre: server umount lustre-OST0001 complete [ 3122.148981] LustreError: 214354:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3122.158050] LustreError: 214354:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 27 previous similar messages [ 3128.616644] Lustre: server umount lustre-OST0000 complete [ 3136.564731] Lustre: server umount lustre-MDT0000 complete [ 3138.405139] LustreError: 212773:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768143121 with bad export cookie 105236926843137588 [ 3138.411691] LustreError: 212773:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3138.412695] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3138.557571] Lustre: server umount lustre-MDT0001 complete [ 3149.199483] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 09:52:11 (1768143131) [ 3152.959321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3153.083340] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3153.100980] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3153.107109] Lustre: Skipped 5 previous similar messages [ 3154.834343] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3158.705329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3158.754803] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3158.768277] Lustre: 218845:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3160.570562] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3162.928192] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3164.695872] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3167.503070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3167.584072] Lustre: Permanently deactivating lustre-OST0000 [ 3167.588340] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3168.909575] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3170.152279] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3172.204940] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3175.630626] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3176.372968] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3178.094484] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3180.621879] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3181.456559] Lustre: Permanently reactivating lustre-OST0000 [ 3184.207093] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3184.212376] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3184.219558] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3184.249504] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3184.260575] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3195.875595] LustreError: lustre-OST0001-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 3195.879308] LustreError: Skipped 5 previous similar messages [ 3199.308116] Lustre: server umount lustre-OST0001 complete [ 3215.327142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3215.416739] Lustre: server umount lustre-OST0000 complete [ 3223.379967] Lustre: server umount lustre-MDT0000 complete [ 3225.050655] LustreError: 218837:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768143207 with bad export cookie 105236926843139219 [ 3225.055059] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3225.059336] LustreError: 218837:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3225.184378] Lustre: server umount lustre-MDT0001 complete [ 3229.346318] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3230.923469] Key type lgssc unregistered [ 3231.084808] LNet: 223248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3231.089474] LNetError: 223248:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3231.097724] LNet: Removed LNI 192.168.203.104@tcp [ 3231.560505] Key type .llcrypt unregistered [ 3231.562727] Key type ._llcrypt unregistered [ 3242.136298] Key type ._llcrypt registered [ 3242.137765] Key type .llcrypt registered [ 3242.189513] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3242.524580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3242.568868] alg: No test for adler32 (adler32-zlib) [ 3243.423858] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 3243.516688] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3245.103207] Key type lgssc registered [ 3245.646555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3250.639692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3250.652714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3251.906434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3253.839733] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3257.973592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3258.037215] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3258.056259] Lustre: 224663:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3258.066165] Lustre: Permanently deactivating lustre-MDT0001 [ 3258.077115] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3258.263098] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3258.759397] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3258.763671] LustreError: 224704:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3258.769237] LustreError: 224704:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3258.774504] LustreError: 224704:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 1, failed: rc = -108 [ 3260.121739] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3262.677685] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3264.426716] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3267.158280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3267.355860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3268.393880] LustreError: 226291:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3268.400499] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3268.401914] LustreError: 226291:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 2 previous similar messages [ 3270.073519] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3272.735236] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3273.700436] LustreError: 226291:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3276.144405] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3276.273270] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3277.299897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3277.303158] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3277.308216] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3279.017903] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3281.784222] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3283.752819] Lustre: Permanently reactivating lustre-MDT0001 [ 3283.763119] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3283.767229] Lustre: Skipped 2 previous similar messages [ 3291.271441] Lustre: lustre-MDT0001: Client dbd78560-80f9-4d2b-8484-473dcd2cae94 (at 192.168.203.4@tcp) reconnecting [ 3293.580375] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3294.178476] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3296.337097] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3296.405919] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3298.154574] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3298.258782] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3299.529800] Lustre: Permanently deactivating lustre-MDT0001 [ 3299.540324] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3299.544426] Lustre: Skipped 2 previous similar messages [ 3301.832720] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3309.028228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3309.029136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3309.035130] Lustre: Skipped 3 previous similar messages [ 3309.038423] Lustre: Skipped 2 previous similar messages [ 3314.008351] LustreError: 228782:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3314.042599] Lustre: server umount lustre-MDT0000 complete [ 3314.145410] LustreError: 225402:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3314.152697] LustreError: 225402:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 3 previous similar messages [ 3315.758707] LustreError: 225395:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768143298 with bad export cookie 10779748173367389468 [ 3315.762325] LustreError: 225395:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3315.762903] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3315.836843] LustreError: 228984:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3315.839251] LustreError: 228984:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3315.892085] Lustre: server umount lustre-MDT0001 complete [ 3323.441078] LustreError: 229185:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3323.444370] LustreError: 229185:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3323.463352] Lustre: server umount lustre-OST0000 complete [ 3328.198522] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 09:55:10 (1768143310) [ 3332.967678] LustreError: 230190:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3332.974118] LustreError: 230190:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3333.038467] Lustre: server umount lustre-OST0001 complete [ 3335.545386] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 3339.999589] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3356.582932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3356.685330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3356.688939] Lustre: Skipped 2 previous similar messages [ 3356.700094] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3356.739761] Lustre: lustre-MDT0000: new disk, initializing [ 3356.763082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3356.770581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3358.444405] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3363.313596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3363.368738] Lustre: 232362:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3363.393747] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3363.396253] Lustre: Skipped 1 previous similar message [ 3363.446806] Lustre: lustre-MDT0001: new disk, initializing [ 3363.517237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3363.523424] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3364.805524] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3368.702846] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3369.891311] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3372.001664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3372.109657] Lustre: lustre-OST0000: new disk, initializing [ 3372.111430] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3372.141262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3372.143555] Lustre: Skipped 1 previous similar message [ 3372.661798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3372.665100] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3372.706187] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3374.068106] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3377.112032] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3377.842087] LustreError: 232368:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3383.082039] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3383.197403] Lustre: lustre-OST0001: new disk, initializing [ 3383.200253] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3385.491541] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3387.935083] LustreError: 232368:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3387.942454] LustreError: 232368:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3389.353331] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3390.076856] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3391.511066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3391.521227] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3391.567118] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3398.031079] LustreError: 232368:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3399.822238] Lustre: lustre-OST0001: Not available for connect from 192.168.203.4@tcp (stopping) [ 3401.703994] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3401.705141] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3401.714100] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3401.717579] LustreError: Skipped 1 previous similar message [ 3401.724024] Lustre: Skipped 1 previous similar message [ 3404.914167] LustreError: 235846:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3404.918409] LustreError: 235846:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3404.932399] LustreError: 234201:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3404.956232] Lustre: server umount lustre-OST0001 complete [ 3411.938938] LustreError: 235234:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3411.944539] LustreError: 235234:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 3 previous similar messages [ 3419.104821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3419.104866] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3419.108557] LustreError: Skipped 1 previous similar message [ 3419.109349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3419.114134] Lustre: Skipped 2 previous similar messages [ 3422.184297] LustreError: 234201:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3422.193391] LustreError: 234201:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 3 previous similar messages [ 3423.540709] LustreError: 236069:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3423.545449] LustreError: 236069:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3423.605634] Lustre: server umount lustre-OST0000 complete [ 3427.299416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3427.301491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3427.307033] Lustre: Skipped 1 previous similar message [ 3427.310658] Lustre: Skipped 3 previous similar messages [ 3431.734532] Lustre: server umount lustre-MDT0000 complete [ 3433.122604] LustreError: 232354:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768143416 with bad export cookie 10779748173367392338 [ 3433.127226] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3439.300460] Lustre: server umount lustre-MDT0001 complete [ 3443.645672] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3444.922428] Key type lgssc unregistered [ 3445.065574] LNet: 237004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3445.069788] LNetError: 237004:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3445.079453] LNet: Removed LNI 192.168.203.104@tcp [ 3445.469066] Key type .llcrypt unregistered [ 3445.470306] Key type ._llcrypt unregistered [ 3461.940139] Key type ._llcrypt registered [ 3461.941136] Key type .llcrypt registered [ 3461.977265] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3462.535082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3462.546676] alg: No test for adler32 (adler32-zlib) [ 3463.402339] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 3463.492805] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3465.087241] Key type lgssc registered [ 3465.632159] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3469.240346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3469.251488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3470.384697] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3470.399383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3470.477926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3472.046597] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3476.327092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3476.370106] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3476.389569] Lustre: 239168:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3476.542299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3478.156729] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3480.875680] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3482.566684] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3486.030138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3486.132696] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3486.222570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3488.433392] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3490.934324] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3491.861399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3537.894447] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3537.897227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3537.898710] LustreError: Skipped 1 previous similar message [ 3537.911984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3537.916825] Lustre: Skipped 1 previous similar message [ 3541.616935] LustreError: 241520:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3541.704295] Lustre: server umount lustre-OST0000 complete [ 3543.800386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3549.933091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3550.127585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3552.935607] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3555.632286] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3557.288404] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3557.292278] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3557.292435] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3557.297934] Lustre: Skipped 1 previous similar message [ 3593.188093] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3593.188271] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3593.191512] LustreError: Skipped 1 previous similar message [ 3593.197253] Lustre: Skipped 1 previous similar message [ 3593.198327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3593.204978] Lustre: Skipped 1 previous similar message [ 3595.768816] LustreError: 243180:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3595.774177] LustreError: 243180:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3595.839025] Lustre: server umount lustre-OST0000 complete [ 3598.306500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3598.307751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3598.314601] Lustre: Skipped 2 previous similar messages [ 3603.342521] LustreError: 243381:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3603.346240] LustreError: 243381:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3603.380975] Lustre: server umount lustre-MDT0000 complete [ 3604.707229] LustreError: 239894:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768143587 with bad export cookie 3864685326903050412 [ 3604.710395] LustreError: 239894:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3604.710597] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3604.820887] Lustre: server umount lustre-MDT0001 complete [ 3608.453191] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3609.901880] Key type lgssc unregistered [ 3610.039757] LNet: 244114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3610.045406] LNetError: 244114:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3610.053598] LNet: Removed LNI 192.168.203.104@tcp [ 3610.486068] Key type .llcrypt unregistered [ 3610.488260] Key type ._llcrypt unregistered [ 3620.917367] Key type ._llcrypt registered [ 3620.918774] Key type .llcrypt registered [ 3620.955988] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3621.295414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3621.329890] alg: No test for adler32 (adler32-zlib) [ 3622.170212] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 3622.247676] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3623.823199] Key type lgssc registered [ 3624.339047] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3628.080346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3628.091597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3629.321427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3631.032968] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3635.227380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3635.397456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3636.938519] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3639.182187] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3640.854818] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3643.833735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3644.021693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3646.585498] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3648.916429] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3654.247492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3664.869647] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3664.870144] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3664.871295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3664.878596] Lustre: Skipped 1 previous similar message [ 3669.874137] LustreError: 248502:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3669.935227] Lustre: server umount lustre-OST0000 complete [ 3671.520100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3671.522910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3671.527755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3671.529860] Lustre: Skipped 1 previous similar message [ 3675.105691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3675.106847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3677.566705] LustreError: 248704:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3677.569585] LustreError: 248704:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3677.600514] Lustre: server umount lustre-MDT0000 complete [ 3679.129746] LustreError: 245468:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768143662 with bad export cookie 15735768124805719723 [ 3679.138356] LustreError: 245468:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3679.138386] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3679.230211] LustreError: 248916:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3679.234465] LustreError: 248916:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3679.318854] Lustre: server umount lustre-MDT0001 complete [ 3683.937859] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3685.232394] Key type lgssc unregistered [ 3685.406787] LNet: 249448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3685.411860] LNetError: 249448:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3685.424556] LNet: Removed LNI 192.168.203.104@tcp [ 3685.829819] Key type .llcrypt unregistered [ 3685.831205] Key type ._llcrypt unregistered [ 3694.460524] Key type ._llcrypt registered [ 3694.461949] Key type .llcrypt registered [ 3694.507027] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3694.854369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3694.892112] alg: No test for adler32 (adler32-zlib) [ 3695.745154] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 3695.839594] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3697.431228] Key type lgssc registered [ 3697.969508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3701.461400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3701.472532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3702.688479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3704.137658] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3707.899072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3708.067854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3709.745019] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3712.222838] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3714.068424] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3717.450416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3717.605718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3719.776665] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3722.099835] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3727.858623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3738.085500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3738.086955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3738.088365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3738.094243] Lustre: Skipped 1 previous similar message [ 3743.205222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3743.209024] Lustre: Skipped 2 previous similar messages [ 3744.179134] LustreError: 253036:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3744.235881] Lustre: server umount lustre-OST0000 complete [ 3748.322396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3748.324396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3748.329793] Lustre: Skipped 1 previous similar message [ 3748.335988] Lustre: Skipped 1 previous similar message [ 3752.015214] LustreError: 253238:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3752.019229] LustreError: 253238:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3752.052981] Lustre: server umount lustre-MDT0000 complete [ 3753.441288] LustreError: 250627:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3753.451122] LustreError: 250627:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 3 previous similar messages [ 3753.781905] LustreError: 252243:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768143736 with bad export cookie 8931804729749345896 [ 3753.786463] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3753.788645] LustreError: 252243:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3753.874264] LustreError: 253440:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3753.876365] LustreError: 253440:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3753.923335] Lustre: server umount lustre-MDT0001 complete [ 3758.552847] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3759.576642] Key type lgssc unregistered [ 3759.696323] LNet: 253971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3759.698424] LNetError: 253971:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3759.708555] LNet: Removed LNI 192.168.203.104@tcp [ 3759.996892] Key type .llcrypt unregistered [ 3759.998850] Key type ._llcrypt unregistered [ 3771.186722] Key type ._llcrypt registered [ 3771.188692] Key type .llcrypt registered [ 3771.229361] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3771.629946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3771.646740] alg: No test for adler32 (adler32-zlib) [ 3772.516787] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 3772.603541] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3774.191183] Key type lgssc registered [ 3774.731613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3778.196417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3778.208432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3779.401912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3781.144613] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3784.755907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3784.957266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3786.558603] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3789.049306] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3790.774343] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3793.982803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3794.143842] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3796.656876] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3799.063344] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3805.420388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3821.027094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3821.029603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3821.031742] LustreError: Skipped 1 previous similar message [ 3821.032466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3821.039166] Lustre: Skipped 1 previous similar message [ 3825.266184] LustreError: 258383:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3825.320949] Lustre: server umount lustre-OST0000 complete [ 3831.266452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3831.266561] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3831.267178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3831.267184] Lustre: Skipped 1 previous similar message [ 3831.283716] Lustre: Skipped 1 previous similar message [ 3832.909964] LustreError: 258585:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3832.914145] LustreError: 258585:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3832.953826] Lustre: server umount lustre-MDT0000 complete [ 3834.618290] LustreError: 255327:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768143817 with bad export cookie 12108915000613277846 [ 3834.624160] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3834.626048] LustreError: 255327:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3834.726532] LustreError: 258787:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3834.729212] LustreError: 258787:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3834.782219] Lustre: server umount lustre-MDT0001 complete [ 3839.284614] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3840.218356] Key type lgssc unregistered [ 3840.333393] LNet: 259317:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3840.335879] LNetError: 259317:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3840.345320] LNet: Removed LNI 192.168.203.104@tcp [ 3840.603802] Key type .llcrypt unregistered [ 3840.604718] Key type ._llcrypt unregistered [ 3848.450178] Key type ._llcrypt registered [ 3848.451621] Key type .llcrypt registered [ 3848.500332] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3849.087543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3849.098037] alg: No test for adler32 (adler32-zlib) [ 3849.977083] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 3850.061149] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3851.639120] Key type lgssc registered [ 3852.030771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3854.835510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3854.845409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3856.038183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3857.443418] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3860.151820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3860.276549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3861.666782] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3863.518451] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3864.656800] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3866.976239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3867.101879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3869.123701] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3870.867769] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3880.423043] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3890.658288] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3890.663206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3890.663288] LustreError: Skipped 1 previous similar message [ 3890.672304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3890.677946] Lustre: Skipped 1 previous similar message [ 3891.763178] LustreError: 262945:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3891.819860] Lustre: server umount lustre-OST0000 complete [ 3895.778672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3895.779966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3895.787656] Lustre: Skipped 2 previous similar messages [ 3899.726232] LustreError: 263147:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3899.730262] LustreError: 263147:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3899.768234] Lustre: server umount lustre-MDT0000 complete [ 3900.896826] LustreError: 260526:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3900.905277] LustreError: 260526:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 3 previous similar messages [ 3901.620390] LustreError: 260479:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768143884 with bad export cookie 179861751027483356 [ 3901.623189] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3901.624748] LustreError: 260479:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3901.705034] LustreError: 263349:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3901.707213] LustreError: 263349:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3901.766729] Lustre: server umount lustre-MDT0001 complete [ 3906.705282] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 3908.215898] Key type lgssc unregistered [ 3908.378629] LNet: 263880:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3908.383636] LNetError: 263880:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3908.394692] LNet: Removed LNI 192.168.203.104@tcp [ 3908.884176] Key type .llcrypt unregistered [ 3908.886257] Key type ._llcrypt unregistered [ 3912.105807] Key type ._llcrypt registered [ 3912.106805] Key type .llcrypt registered [ 3912.141992] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 3917.186256] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 3923.040566] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 3923.364514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3923.399497] alg: No test for adler32 (adler32-zlib) [ 3924.303316] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 3924.423890] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 3926.023187] Key type lgssc registered [ 3926.624158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3943.571750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3943.585320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3944.724116] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3944.739823] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3944.796816] Lustre: lustre-MDT0000: new disk, initializing [ 3944.835518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3944.844264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3946.638144] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3952.287362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3952.352861] Lustre: 267351:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3952.394077] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3952.397222] Lustre: Skipped 1 previous similar message [ 3952.439240] Lustre: lustre-MDT0001: new disk, initializing [ 3952.483645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3952.501176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3952.505553] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3954.076422] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3957.878977] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3959.648869] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3962.723522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3962.874321] Lustre: lustre-OST0000: new disk, initializing [ 3962.876375] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3962.907270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3965.147148] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 3966.926919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3966.933690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3966.998963] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3968.882821] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3970.637614] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3970.752258] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3972.439895] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3972.523676] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3977.188118] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3977.192522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3977.194526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3977.199210] Lustre: Skipped 1 previous similar message [ 3979.372187] LustreError: 270281:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3979.421147] Lustre: server umount lustre-OST0000 complete [ 3982.306047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3982.306518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3982.310614] Lustre: Skipped 1 previous similar message [ 3982.312571] Lustre: Skipped 1 previous similar message [ 3987.028655] LustreError: 270482:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3987.032947] LustreError: 270482:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3987.082493] Lustre: server umount lustre-MDT0000 complete [ 3988.962219] LustreError: 267344:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768143971 with bad export cookie 8699546635163415887 [ 3988.967875] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3989.068584] LustreError: 270684:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3989.070576] LustreError: 270684:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3989.132337] Lustre: server umount lustre-MDT0001 complete [ 3993.445611] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:06:15 (1768143975) [ 3996.835675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3997.068942] LustreError: 271353:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3997.079050] LustreError: 271353:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 3 previous similar messages [ 3997.104709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3999.094697] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4002.274883] LustreError: 271353:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4003.387979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4003.572091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4005.469097] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4008.048981] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4009.780312] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4013.003967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4013.171554] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4015.572833] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4017.707597] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4026.850058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4026.853639] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4026.855034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4026.855040] Lustre: Skipped 1 previous similar message [ 4026.857550] Lustre: Skipped 1 previous similar message [ 4026.862650] LustreError: Skipped 1 previous similar message [ 4031.970267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4031.973996] Lustre: Skipped 2 previous similar messages [ 4032.114191] LustreError: 273598:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4032.120941] LustreError: 273598:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4032.182550] Lustre: server umount lustre-OST0000 complete [ 4034.528747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4034.536232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4037.089488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4037.091788] Lustre: Skipped 2 previous similar messages [ 4047.329752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4047.332784] Lustre: Skipped 3 previous similar messages [ 4048.351215] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4048.409917] LustreError: 273800:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4048.413526] LustreError: 273800:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4048.453380] Lustre: server umount lustre-MDT0000 complete [ 4050.174608] LustreError: 271332:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768144033 with bad export cookie 8699546635163416769 [ 4050.178836] LustreError: 272074:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4050.179998] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4050.180160] LustreError: 271332:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4050.297304] Lustre: server umount lustre-MDT0001 complete [ 4054.310244] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 4055.504201] Key type lgssc unregistered [ 4055.689629] LNet: 274534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4055.694474] LNetError: 274534:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4055.706703] LNet: Removed LNI 192.168.203.104@tcp [ 4056.207534] Key type .llcrypt unregistered [ 4056.209731] Key type ._llcrypt unregistered [ 4067.940597] Key type ._llcrypt registered [ 4067.941849] Key type .llcrypt registered [ 4067.990214] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 4075.495280] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4075.888897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4075.938833] alg: No test for adler32 (adler32-zlib) [ 4076.855175] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 4076.966925] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 4078.575196] Key type lgssc registered [ 4079.175755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4100.642521] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4100.654044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4101.771436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4101.786845] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4101.828394] Lustre: lustre-MDT0000: new disk, initializing [ 4101.861872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4101.870228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4103.643528] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4109.426413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4109.498204] Lustre: 278295:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4109.541250] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4109.544676] Lustre: Skipped 1 previous similar message [ 4109.594222] Lustre: lustre-MDT0001: new disk, initializing [ 4109.635291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4109.651893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4109.657127] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4111.430781] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4115.543798] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4117.229216] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4120.079717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4120.266972] Lustre: lustre-OST2710: new disk, initializing [ 4120.270816] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4120.324595] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4123.068307] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4127.235065] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4129.295517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4129.301657] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4129.349985] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4130.315286] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4130.432830] Lustre: lustre-OST03e8: new disk, initializing [ 4130.436955] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4130.494327] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4133.072868] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4135.589510] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4135.931454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4135.934976] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4135.959100] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4138.384492] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4138.513662] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4140.456941] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4140.590241] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4142.400445] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4142.537595] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4144.483582] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4144.609909] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4146.149608] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4146.154502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4146.157034] Lustre: Skipped 3 previous similar messages [ 4146.160306] Lustre: Skipped 2 previous similar messages [ 4151.265814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4151.269577] Lustre: Skipped 2 previous similar messages [ 4152.146119] LustreError: 282626:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4152.188554] Lustre: server umount lustre-MDT0000 complete [ 4153.971101] LustreError: 278286:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768144136 with bad export cookie 11761181394602781043 [ 4153.976636] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4153.980917] LustreError: 278286:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4154.073546] LustreError: 282829:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4154.076463] LustreError: 282829:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4154.137411] Lustre: server umount lustre-MDT0001 complete [ 4165.628147] LustreError: 283029:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4165.632914] LustreError: 283029:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4165.666609] Lustre: server umount lustre-OST2710 complete [ 4177.458414] LustreError: 283232:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4177.461459] LustreError: 283232:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4177.506559] Lustre: server umount lustre-OST03e8 complete [ 4185.852300] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 4190.588461] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4207.446856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4207.589960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4207.608749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4207.672754] Lustre: lustre-MDT0000: new disk, initializing [ 4207.711635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4207.723097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4209.428374] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4214.946288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4215.001590] Lustre: 286014:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4215.133053] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4216.864824] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4220.506525] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4222.254623] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4225.476583] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4225.647629] Lustre: lustre-OST0000: new disk, initializing [ 4225.649576] Lustre: Skipped 1 previous similar message [ 4225.652161] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4225.654587] Lustre: Skipped 2 previous similar messages [ 4225.691868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4225.693616] Lustre: Skipped 1 previous similar message [ 4228.380350] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4232.272787] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4232.717141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4232.722247] Lustre: Skipped 1 previous similar message [ 4232.725690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4232.774823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4234.135299] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4234.262137] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4235.903633] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4236.027671] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4237.797200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4237.797881] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4237.806285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4237.809449] LustreError: Skipped 1 previous similar message [ 4237.813137] Lustre: Skipped 1 previous similar message [ 4238.963710] LustreError: 288947:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4238.969416] LustreError: 288947:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4239.023320] Lustre: server umount lustre-OST0000 complete [ 4240.864493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4240.869110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4240.876375] Lustre: Skipped 1 previous similar message [ 4240.879747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4242.914814] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4248.032647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4248.036322] Lustre: Skipped 3 previous similar messages [ 4255.199209] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4255.253939] LustreError: 289147:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4255.257823] LustreError: 289147:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4255.299249] Lustre: server umount lustre-MDT0000 complete [ 4257.034755] LustreError: 287756:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768144239 with bad export cookie 11761181394602782716 [ 4257.039498] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4257.040220] LustreError: 287756:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4257.175521] Lustre: server umount lustre-MDT0001 complete [ 4261.102972] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 10:10:43 (1768144243) [ 4261.912824] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4262.801985] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 10:10:44 (1768144244) [ 4264.124341] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4275.662808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4275.784558] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4275.794914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4275.877562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4277.561662] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4281.187375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4281.225279] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4281.236063] Lustre: 291445:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4282.917777] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4285.421155] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4287.163116] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4301.144908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4301.266112] Lustre: Denying initial registration attempt from nid 192.168.203.104@tcp, specified as failover [ 4301.271581] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4301.277374] LustreError: 293056:0:(tgt_mount.c:2483:server_fill_super()) Unable to start targets: -99 [ 4301.281993] LustreError: 293056:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4301.285891] LustreError: 293056:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4301.318926] Lustre: server umount lustre-OST0000 complete [ 4301.321198] LustreError: 293056:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4305.377723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4305.378913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4305.385322] Lustre: Skipped 1 previous similar message [ 4305.389073] Lustre: Skipped 2 previous similar messages [ 4310.533128] LustreError: 293423:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4310.536604] LustreError: 293423:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4310.559844] Lustre: server umount lustre-MDT0000 complete [ 4312.232851] LustreError: 291436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768144295 with bad export cookie 11761181394602783584 [ 4312.237629] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4312.238892] LustreError: 291436:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4312.386294] Lustre: server umount lustre-MDT0001 complete [ 4317.060191] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 4318.598287] Key type lgssc unregistered [ 4318.778716] LNet: 294157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4318.784135] LNetError: 294157:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4318.796684] LNet: Removed LNI 192.168.203.104@tcp [ 4319.275332] Key type .llcrypt unregistered [ 4319.277774] Key type ._llcrypt unregistered [ 4323.545700] Key type ._llcrypt registered [ 4323.547682] Key type .llcrypt registered [ 4323.604747] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4323.960824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4324.009753] alg: No test for adler32 (adler32-zlib) [ 4324.872983] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 4324.976794] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 4326.583199] Key type lgssc registered [ 4327.148837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4366.801774] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4370.014441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4370.026190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4371.152688] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4371.160623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4371.224096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4372.703066] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4376.691326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4376.751329] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4376.774578] Lustre: 296868:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4376.922077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4378.610989] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4381.158688] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4382.927138] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4397.211506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4397.344549] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4397.455098] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4399.702731] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4402.168505] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4403.238353] LustreError: 299136:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4403.293916] Lustre: server umount lustre-OST0000 complete [ 4406.266875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4406.268538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4406.275623] Lustre: Skipped 1 previous similar message [ 4411.087121] LustreError: 299336:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4411.090919] LustreError: 299336:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4411.122825] Lustre: server umount lustre-MDT0000 complete [ 4412.761785] LustreError: 296859:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768144395 with bad export cookie 3915011998322321278 [ 4412.767222] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4412.768636] LustreError: 296859:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4412.876107] LustreError: 299541:0:(obd_class.h:479:obd_check_dev()) Device 7 not setup [ 4412.879500] LustreError: 299541:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4412.952783] Lustre: server umount lustre-MDT0001 complete [ 4417.346649] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 4418.853137] Key type lgssc unregistered [ 4419.010548] LNet: 300073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4419.015468] LNetError: 300073:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4419.024462] LNet: Removed LNI 192.168.203.104@tcp [ 4419.447079] Key type .llcrypt unregistered [ 4419.449873] Key type ._llcrypt unregistered [ 4431.725370] Key type ._llcrypt registered [ 4431.726608] Key type .llcrypt registered [ 4431.771419] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4432.082839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4432.124026] alg: No test for adler32 (adler32-zlib) [ 4432.983565] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 4433.066153] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 4434.639220] Key type lgssc registered [ 4435.081908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4438.831708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4438.841829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4440.027757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4441.722885] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4445.534449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4445.690173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4447.588287] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4450.094320] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4451.527189] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4454.811305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4455.004631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4457.462466] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4459.985458] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4469.731593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4469.733772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4469.738954] Lustre: Skipped 2 previous similar messages [ 4469.742353] Lustre: Skipped 1 previous similar message [ 4470.413616] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 4470.417936] Lustre: Skipped 1 previous similar message [ 4473.868579] LustreError: 303693:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4473.911584] Lustre: server umount lustre-MDT0000 complete [ 4474.849292] LustreError: 301448:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4474.857774] LustreError: 301448:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 4 previous similar messages [ 4475.522993] LustreError: 302557:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4475.669049] LustreError: 301429:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768144458 with bad export cookie 4277273240626618719 [ 4475.671753] LustreError: 301429:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4475.672164] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4479.967928] LustreError: 302557:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4479.968215] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4479.981928] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4481.809460] LustreError: 303897:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4481.813466] LustreError: 303897:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4481.872684] Lustre: server umount lustre-MDT0001 complete [ 4485.196138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4489.436265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4501.472484] LustreError: 300757:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007345cf2b x1854033798127104/t0(0) o250->MGC192.168.203.104@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4501.627177] LustreError: 303054:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4501.633649] LustreError: 303054:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 2 previous similar messages [ 4501.671989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4501.705982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4503.319242] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4506.786694] LustreError: 304705:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4506.795445] LustreError: 304705:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 1 previous similar message [ 4507.450320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4507.652680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4509.415560] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4511.783996] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4512.072815] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4517.074331] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 4518.809357] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4521.503531] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 2 sec [ 4527.078943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4527.079981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4527.083112] Lustre: Skipped 1 previous similar message [ 4527.088255] Lustre: Skipped 3 previous similar messages [ 4529.270223] LustreError: 306198:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4529.274354] LustreError: 306198:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4529.344619] Lustre: server umount lustre-OST0000 complete [ 4532.193881] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4532.194865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4532.198597] Lustre: Skipped 1 previous similar message [ 4537.366780] LustreError: 306400:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4537.370149] LustreError: 306400:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4537.415095] Lustre: server umount lustre-MDT0000 complete [ 4539.303737] LustreError: 301472:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768144522 with bad export cookie 4277273240626625873 [ 4539.311328] LustreError: 301472:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4539.311683] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4539.481117] Lustre: server umount lustre-MDT0001 complete [ 4544.307099] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 4545.692800] Key type lgssc unregistered [ 4545.813342] LNet: 307135:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4545.816417] LNetError: 307135:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4545.826285] LNet: Removed LNI 192.168.203.104@tcp [ 4546.129119] Key type .llcrypt unregistered [ 4546.130681] Key type ._llcrypt unregistered [ 4558.469580] Key type ._llcrypt registered [ 4558.471178] Key type .llcrypt registered [ 4558.512468] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4558.807363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4558.814882] alg: No test for adler32 (adler32-zlib) [ 4559.690555] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 4559.796558] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 4561.391162] Key type lgssc registered [ 4561.908035] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4565.181713] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4565.191805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4566.358411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4568.041857] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4572.268749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4572.431990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4574.204896] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4576.910924] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4578.696831] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4581.857799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4581.858618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4581.864524] Lustre: Skipped 1 previous similar message [ 4582.881610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4582.885449] Lustre: Skipped 2 previous similar messages [ 4586.252674] LustreError: 309896:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4586.293088] Lustre: server umount lustre-MDT0000 complete [ 4588.002266] LustreError: 308508:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4588.006678] LustreError: 309223:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768144570 with bad export cookie 3791848277590064823 [ 4588.012498] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4588.013656] LustreError: 308508:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 3 previous similar messages [ 4588.020631] LustreError: 309223:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4588.105743] LustreError: 310100:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4588.108596] LustreError: 310100:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4588.162084] Lustre: server umount lustre-MDT0001 complete [ 4595.163362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4595.319309] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4595.338405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4595.448926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4597.231631] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4600.932061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4600.976863] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4600.987939] Lustre: 310868:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4601.097574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4602.470125] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4604.861403] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4606.229909] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4609.730712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4609.820678] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4609.827369] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4609.831062] LustreError: 312526:0:(tgt_mount.c:2483:server_fill_super()) Unable to start targets: -2 [ 4609.834098] LustreError: 312526:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4609.836724] LustreError: 312526:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4609.867771] Lustre: server umount lustre-OST0000 complete [ 4609.870111] LustreError: 312526:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4614.079061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4614.182147] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4614.290001] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4616.961108] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4619.420464] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4623.009488] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4623.062636] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4623.105767] Lustre: lustre-OST0001: new disk, initializing [ 4623.109492] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4623.165164] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4623.387804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4623.399545] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4623.427234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4623.452905] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4625.458370] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4627.577447] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4633.575885] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4633.579410] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4633.581845] LustreError: Skipped 1 previous similar message [ 4633.589746] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4633.594718] Lustre: Skipped 1 previous similar message [ 4638.695083] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4638.698956] Lustre: Skipped 1 previous similar message [ 4643.295122] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4643.319136] LustreError: 314633:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 4643.323044] LustreError: 314633:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4643.354458] Lustre: server umount lustre-OST0001 complete [ 4643.819856] LustreError: 312990:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4643.830110] LustreError: 312990:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 1 previous similar message [ 4648.931480] LustreError: 312991:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4648.931522] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4648.932696] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4648.932703] Lustre: Skipped 1 previous similar message [ 4648.933752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4648.941109] LustreError: 312991:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 1 previous similar message [ 4648.944261] LustreError: Skipped 1 previous similar message [ 4651.187827] LustreError: 314837:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4651.192113] LustreError: 314837:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4651.251145] Lustre: server umount lustre-OST0000 complete [ 4654.050787] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4654.059323] Lustre: Skipped 2 previous similar messages [ 4659.168966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4659.173227] Lustre: Skipped 4 previous similar messages [ 4667.359195] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4667.421567] LustreError: 315039:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4667.424933] LustreError: 315039:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4667.456669] Lustre: server umount lustre-MDT0000 complete [ 4668.385293] LustreError: 310879:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4668.395922] LustreError: 310879:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 1 previous similar message [ 4669.103841] LustreError: 310860:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768144652 with bad export cookie 3791848277590065362 [ 4669.110519] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4669.111053] LustreError: 310860:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4669.272571] Lustre: server umount lustre-MDT0001 complete [ 4674.205738] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 4675.475344] Key type lgssc unregistered [ 4675.625849] LNet: 315773:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4675.631653] LNetError: 315773:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4675.640564] LNet: Removed LNI 192.168.203.104@tcp [ 4676.011988] Key type .llcrypt unregistered [ 4676.014244] Key type ._llcrypt unregistered [ 4712.250146] Key type ._llcrypt registered [ 4712.251369] Key type .llcrypt registered [ 4712.284656] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 4718.657782] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4719.097913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4719.106752] alg: No test for adler32 (adler32-zlib) [ 4719.951590] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 4720.037599] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 4721.615128] Key type lgssc registered [ 4721.916207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4733.417904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4733.427048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4734.547239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4734.562493] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4734.622195] Lustre: lustre-MDT0000: new disk, initializing [ 4734.649278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4734.657198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4735.962650] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4740.927982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4740.994207] Lustre: 320639:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4741.031424] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4741.034715] Lustre: Skipped 1 previous similar message [ 4741.111356] Lustre: lustre-MDT0001: new disk, initializing [ 4741.151365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4741.167999] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4741.172260] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4742.405088] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4745.577942] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4746.858674] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4749.191803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4749.343676] Lustre: lustre-OST0000: new disk, initializing [ 4749.346118] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4749.393387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4751.842044] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4753.923349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4753.927464] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4753.963700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4755.275640] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4756.925949] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4757.032526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4758.239381] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4758.306758] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4759.012480] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4759.014672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4759.016293] LustreError: Skipped 1 previous similar message [ 4759.016677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4759.022217] Lustre: Skipped 1 previous similar message [ 4764.129635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4764.133016] Lustre: Skipped 2 previous similar messages [ 4769.253258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4769.257204] Lustre: Skipped 1 previous similar message [ 4773.343176] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4773.366075] LustreError: 323567:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4773.406157] Lustre: server umount lustre-OST0000 complete [ 4776.928461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4776.933540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4776.943089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4779.488793] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4784.608926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4784.612599] Lustre: Skipped 3 previous similar messages [ 4789.215197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4789.276988] LustreError: 323770:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4789.280459] LustreError: 323770:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4789.314788] Lustre: server umount lustre-MDT0000 complete [ 4791.125041] LustreError: 322379:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768144774 with bad export cookie 4721011913443743743 [ 4791.131978] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4791.132058] LustreError: 322379:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4791.227632] LustreError: 323973:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4791.230964] LustreError: 323973:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4791.305680] Lustre: server umount lustre-MDT0001 complete [ 4795.500991] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 10:19:37 (1768144777) [ 4799.237861] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 10:19:41 (1768144781) [ 4803.129732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4803.366247] LustreError: 324987:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4803.373455] LustreError: 324987:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 3 previous similar messages [ 4803.401305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4805.085532] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4808.673072] LustreError: 324987:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4809.320525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4809.513468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4811.298390] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4813.865817] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4815.471622] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4818.760778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4818.942928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4821.520205] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4823.951266] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4832.994485] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4833.915590] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4834.651896] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4835.391459] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4836.336683] Lustre: Failing over lustre-MDT0000 [ 4836.591739] LustreError: 327422:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4836.593940] LustreError: 327422:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4836.619177] Lustre: server umount lustre-MDT0000 complete [ 4838.371728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4838.373712] LustreError: 324981:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4838.378940] Lustre: Skipped 2 previous similar messages [ 4838.388250] LustreError: 324981:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 1 previous similar message [ 4843.489098] LustreError: 326101:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4843.497830] LustreError: 326101:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 4 previous similar messages [ 4848.610169] LustreError: 326101:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4848.614890] LustreError: 326101:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 3 previous similar messages [ 4850.479839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4850.528926] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4850.655035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4850.695458] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4852.407748] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4854.914841] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4855.797899] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4855.809961] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4855.840735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4857.374500] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4858.121253] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4860.271646] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4861.233365] Lustre: Failing over lustre-MDT0000 [ 4861.499283] LustreError: 328501:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4861.501417] LustreError: 328501:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4861.526692] Lustre: server umount lustre-MDT0000 complete [ 4865.164815] LustreError: 324981:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.4@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4865.178705] LustreError: 324981:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 1 previous similar message [ 4866.019610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4866.020938] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4866.026413] Lustre: Skipped 1 previous similar message [ 4875.366556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4875.449699] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4875.624749] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4877.254543] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4879.000057] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4880.870059] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4880.872347] Lustre: Skipped 2 previous similar messages [ 4880.876280] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4880.892218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4882.252764] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4882.881587] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4885.987956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4885.988268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4885.989708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4885.995201] Lustre: Skipped 2 previous similar messages [ 4891.544798] LustreError: 329527:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4891.548584] LustreError: 329527:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4891.583854] Lustre: server umount lustre-MDT0000 complete [ 4893.233797] LustreError: 324966:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768144876 with bad export cookie 4721011913443746396 [ 4893.238076] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4893.238443] LustreError: 324966:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4893.379729] Lustre: server umount lustre-MDT0001 complete [ 4905.519038] Lustre: server umount lustre-OST0000 complete [ 4911.186846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4911.324472] LustreError: 330757:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4911.329417] LustreError: 330757:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 8 previous similar messages [ 4911.344270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4911.345975] Lustre: Skipped 1 previous similar message [ 4912.626876] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4916.147843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4917.894690] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4920.326424] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4922.132210] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4924.845313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4927.462943] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4930.081140] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4937.324107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 4942.820736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4942.822739] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4942.826654] Lustre: Skipped 1 previous similar message [ 4942.827741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4942.829885] LustreError: Skipped 1 previous similar message [ 4942.831875] Lustre: Skipped 5 previous similar messages [ 4943.600239] LustreError: 333019:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4943.604341] LustreError: 333019:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4943.663410] Lustre: server umount lustre-OST0000 complete [ 4947.626827] Lustre: server umount lustre-MDT0000 complete [ 4947.936799] LustreError: 330756:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4947.946163] LustreError: 330756:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 1 previous similar message [ 4949.385996] LustreError: 332919:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768144932 with bad export cookie 4721011913443747124 [ 4949.390949] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4949.392259] LustreError: 332919:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4949.528328] Lustre: server umount lustre-MDT0001 complete [ 4954.314500] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 4955.608347] Key type lgssc unregistered [ 4955.766567] LNet: 333953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4955.771394] LNetError: 333953:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4955.783575] LNet: Removed LNI 192.168.203.104@tcp [ 4956.249379] Key type .llcrypt unregistered [ 4956.251661] Key type ._llcrypt unregistered [ 4968.368994] Key type ._llcrypt registered [ 4968.370217] Key type .llcrypt registered [ 4968.412784] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 4968.724896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4968.751075] alg: No test for adler32 (adler32-zlib) [ 4969.603537] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 4969.690913] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 4971.279202] Key type lgssc registered [ 4971.815253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4975.147532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4975.154778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4976.326946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4977.885860] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4981.310687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4981.432604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4982.654004] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4984.466706] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4986.206752] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4989.460701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4989.609594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4992.052192] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 4994.455359] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5001.902961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5012.453040] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5012.456937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5012.459055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5012.459214] Lustre: Skipped 2 previous similar messages [ 5015.427568] LustreError: 337689:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5015.460205] Lustre: server umount lustre-MDT0000 complete [ 5017.185733] LustreError: 335407:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768145000 with bad export cookie 11802159907131591107 [ 5017.192898] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5017.289397] LustreError: 337892:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5017.293352] LustreError: 337892:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5017.357103] Lustre: server umount lustre-MDT0001 complete [ 5029.496199] LustreError: 338092:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5029.500188] LustreError: 338092:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5029.566436] Lustre: server umount lustre-OST0000 complete [ 5035.813688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5036.032158] LustreError: 338820:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5036.039429] LustreError: 338820:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 2 previous similar messages [ 5036.060249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5037.805464] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5041.121519] LustreError: 338820:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5041.709987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5041.863879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5043.436460] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5045.980195] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5047.329638] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5049.525668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5051.627937] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5053.665787] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5060.969616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5072.864031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5072.867132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5072.872754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5072.875224] Lustre: Skipped 2 previous similar messages [ 5076.449454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5076.450045] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5076.451873] Lustre: Skipped 1 previous similar message [ 5076.459573] Lustre: Skipped 1 previous similar message [ 5077.889471] LustreError: 341186:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5077.893519] LustreError: 341186:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5077.941487] Lustre: server umount lustre-MDT0000 complete [ 5079.442064] LustreError: 338800:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768145062 with bad export cookie 11802159907131592206 [ 5079.445340] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5079.446974] LustreError: 338800:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5079.578565] Lustre: server umount lustre-MDT0001 complete [ 5091.056739] LustreError: 341588:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5091.058540] LustreError: 341588:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5091.093431] Lustre: server umount lustre-OST0000 complete [ 5095.948356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5096.110804] LustreError: 342416:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5096.132845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5096.135722] Lustre: Skipped 1 previous similar message [ 5097.278044] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5099.881957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5101.205383] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5102.886499] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5103.968828] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5106.102960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5106.144857] LustreError: 344011:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5107.976399] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5109.290953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5109.665298] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5114.336547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5114.338973] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5114.339571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5114.342884] Lustre: Skipped 1 previous similar message [ 5114.344865] Lustre: Skipped 1 previous similar message [ 5116.844085] LustreError: 344652:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5116.846304] LustreError: 344652:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5116.880647] Lustre: server umount lustre-OST0000 complete [ 5119.456566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5119.457104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5119.459913] Lustre: Skipped 1 previous similar message [ 5119.461471] Lustre: Skipped 1 previous similar message [ 5124.236860] Lustre: server umount lustre-MDT0000 complete [ 5124.575920] LustreError: 342416:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5124.581868] LustreError: 342416:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 1 previous similar message [ 5125.506179] LustreError: 342396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768145108 with bad export cookie 11802159907131593214 [ 5125.508767] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5125.509309] LustreError: 342396:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5125.606976] Lustre: server umount lustre-MDT0001 complete [ 5128.962763] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 5129.918739] Key type lgssc unregistered [ 5130.030385] LNet: 345590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5130.032870] LNetError: 345590:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5130.044277] LNet: Removed LNI 192.168.203.104@tcp [ 5130.321368] Key type .llcrypt unregistered [ 5130.322301] Key type ._llcrypt unregistered [ 5138.830900] Key type ._llcrypt registered [ 5138.832143] Key type .llcrypt registered [ 5138.864894] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5139.237537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5139.258667] alg: No test for adler32 (adler32-zlib) [ 5140.110028] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 5140.192078] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 5141.775162] Key type lgssc registered [ 5142.169644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5144.823105] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5144.828668] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5144.831922] LustreError: 346986:0:(osd_handler.c:8561:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5144.841193] LustreError: 346986:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5144.844206] LustreError: 346986:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5144.847786] LustreError: 346986:0:(tgt_mount.c:2451:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5144.851524] LustreError: 346986:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5147.279178] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5147.281706] LustreError: 347383:0:(osd_handler.c:8561:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5147.289860] LustreError: 347383:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5147.291886] LustreError: 347383:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5147.294015] LustreError: 347383:0:(tgt_mount.c:2451:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5147.296761] LustreError: 347383:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5153.503855] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 5154.527729] Key type lgssc unregistered [ 5154.653303] LNet: 348366:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5154.656375] LNetError: 348366:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5154.662310] LNet: Removed LNI 192.168.203.104@tcp [ 5154.955435] Key type .llcrypt unregistered [ 5154.957102] Key type ._llcrypt unregistered [ 5160.907345] Key type ._llcrypt registered [ 5160.908401] Key type .llcrypt registered [ 5160.954710] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 5167.181247] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5167.466650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5167.482817] alg: No test for adler32 (adler32-zlib) [ 5168.327215] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 5168.396558] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 5169.975112] Key type lgssc registered [ 5170.318639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5181.055739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5181.060476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5182.133833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5182.145150] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5182.177784] Lustre: lustre-MDT0000: new disk, initializing [ 5182.201104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5182.208056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5183.362757] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5187.601807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5187.641759] Lustre: 351459:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5187.665223] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5187.667600] Lustre: Skipped 1 previous similar message [ 5187.704544] Lustre: lustre-MDT0001: new disk, initializing [ 5187.739516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5187.753914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5187.758624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5189.117257] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5192.385229] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5193.634595] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5195.920456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5196.032666] Lustre: lustre-OST0000: new disk, initializing [ 5196.034738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5196.067139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5197.896090] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5200.875779] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5202.244288] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5202.934851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5202.937676] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5202.968945] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5203.333224] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5204.665834] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5204.736836] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5208.032759] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5208.034396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5208.036463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5208.041939] Lustre: Skipped 1 previous similar message [ 5208.045667] Lustre: Skipped 1 previous similar message [ 5211.499087] LustreError: 354393:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5211.538417] Lustre: server umount lustre-OST0000 complete [ 5213.153031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5213.153456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5213.156497] Lustre: Skipped 1 previous similar message [ 5213.159727] Lustre: Skipped 1 previous similar message [ 5218.272118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5218.273748] Lustre: Skipped 1 previous similar message [ 5219.003229] LustreError: 354595:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5219.005144] LustreError: 354595:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5219.032981] Lustre: server umount lustre-MDT0000 complete [ 5220.214202] LustreError: 351448:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768145203 with bad export cookie 7537425169394873411 [ 5220.217122] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5220.217198] LustreError: 351448:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5220.274152] LustreError: 354796:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5220.275925] LustreError: 354796:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5220.325470] Lustre: server umount lustre-MDT0001 complete [ 5223.035251] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5223.598288] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 10:26:45 (1768145205) [ 5225.719727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5225.875596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5227.082921] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5229.869566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5229.992397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5231.286488] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5233.075355] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5234.328562] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5236.477736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5238.342366] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5240.041248] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5242.452378] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5242.528961] Lustre: lustre-OST0001: new disk, initializing [ 5242.531074] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5242.570571] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5242.572846] Lustre: Skipped 1 previous similar message [ 5244.853800] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5245.472344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5245.480041] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5245.529063] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5248.135311] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5250.539737] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5250.541123] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5250.543775] LustreError: Skipped 1 previous similar message [ 5250.549386] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5250.553387] Lustre: Skipped 1 previous similar message [ 5254.792564] Lustre: lustre-OST0001: Not available for connect from 192.168.203.4@tcp (stopping) [ 5254.794200] Lustre: Skipped 2 previous similar messages [ 5256.043095] LustreError: 358704:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5256.044867] LustreError: 358704:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5256.064698] Lustre: server umount lustre-OST0001 complete [ 5256.677471] LustreError: 358433:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5256.681659] LustreError: 358433:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 2 previous similar messages [ 5261.796397] LustreError: 357106:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5261.800467] LustreError: 357106:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 2 previous similar messages [ 5266.916208] LustreError: 357104:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5266.920122] LustreError: 357104:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 1 previous similar message [ 5270.496024] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5270.497973] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5270.500814] Lustre: Skipped 1 previous similar message [ 5270.502352] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5272.035161] LustreError: 358433:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5272.041420] LustreError: 358433:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 1 previous similar message [ 5277.155315] LustreError: 357104:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5277.159285] LustreError: 357104:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 1 previous similar message [ 5282.271076] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5282.293217] LustreError: 358928:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5282.295837] LustreError: 358928:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5282.327427] Lustre: server umount lustre-OST0000 complete [ 5286.367823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5286.369788] LustreError: Skipped 1 previous similar message [ 5286.370892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5286.373898] Lustre: Skipped 1 previous similar message [ 5287.392355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5287.393963] Lustre: Skipped 8 previous similar messages [ 5289.748081] Lustre: server umount lustre-MDT0000 complete [ 5290.920287] LustreError: 355492:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768145273 with bad export cookie 7537425169394874279 [ 5290.923376] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5290.925527] LustreError: 355492:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5305.311203] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5305.347111] LustreError: 359335:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5305.349410] LustreError: 359335:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5305.410430] Lustre: server umount lustre-MDT0001 complete [ 5308.935225] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 5309.923370] Key type lgssc unregistered [ 5310.052310] LNet: 359867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5310.054580] LNetError: 359867:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5310.060292] LNet: Removed LNI 192.168.203.104@tcp [ 5310.343102] Key type .llcrypt unregistered [ 5310.344668] Key type ._llcrypt unregistered [ 5327.133869] Key type ._llcrypt registered [ 5327.134843] Key type .llcrypt registered [ 5327.166697] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5327.444044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5327.457303] alg: No test for adler32 (adler32-zlib) [ 5328.306156] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 5328.382811] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 5329.959129] Key type lgssc registered [ 5330.284869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5332.604535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5332.609431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5333.689670] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5333.696854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5333.757268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5334.887563] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5337.657961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5337.691718] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5337.704563] Lustre: 362621:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5337.817442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5339.140954] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5340.976028] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5342.157389] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5343.200788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5343.201239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5343.206047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5343.208443] Lustre: Skipped 1 previous similar message [ 5348.320670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5348.322355] Lustre: Skipped 1 previous similar message [ 5348.934176] LustreError: 363975:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [ 5348.956824] Lustre: server umount lustre-MDT0000 complete [ 5350.273459] LustreError: 363342:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768145333 with bad export cookie 3085419043061519517 [ 5350.277294] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5350.277367] LustreError: 363342:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5350.342110] LustreError: 364177:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 5350.343956] LustreError: 364177:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5350.398846] Lustre: server umount lustre-MDT0001 complete [ 5353.405177] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 10:28:55 (1768145335) [ 5355.517216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5355.681075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5356.868439] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5359.612163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5359.734459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5361.021481] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5362.797122] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5363.993327] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5366.210681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5366.277945] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5366.348139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5368.160745] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5369.856930] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5373.625558] Lustre: Permanently deactivating lustre-OST0000 [ 5373.630943] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5375.408755] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5376.821893] LustreError: 367444:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5376.824820] LustreError: 367444:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5376.826647] LustreError: 367444:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5377.423081] LustreError: 367492:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5377.425409] LustreError: 367492:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5384.115118] LustreError: 367589:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5384.117022] LustreError: 367589:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5384.147572] Lustre: server umount lustre-OST0000 complete [ 5385.044613] LustreError: 367742:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5385.046598] LustreError: 367742:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5385.695898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5385.698598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5385.701762] Lustre: Skipped 1 previous similar message [ 5385.703796] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5389.281385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5389.282363] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5394.400217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5394.402347] Lustre: Skipped 2 previous similar messages [ 5395.428243] LustreError: 367839:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5395.431068] LustreError: 367839:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5395.457800] Lustre: server umount lustre-MDT0000 complete [ 5396.899346] LustreError: 364873:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768145379 with bad export cookie 3085419043061520014 [ 5396.902537] LustreError: 364873:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5396.902662] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5406.179344] LustreError: 368041:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5406.181509] LustreError: 368041:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5406.231584] Lustre: server umount lustre-MDT0001 complete [ 5409.396924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5410.737403] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5411.604323] LustreError: 368882:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5413.550658] Lustre: 369221:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.104@tcp [ 5414.195378] LustreError: 369318:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5414.197262] LustreError: 369318:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5414.218418] Lustre: server umount MGS complete [ 5417.749887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5417.899696] LustreError: 369944:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5417.903868] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5417.905142] LustreError: 369944:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 2 previous similar messages [ 5417.908901] Lustre: Skipped 1 previous similar message [ 5417.927135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5419.069918] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5421.787120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5421.882096] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5423.156961] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5424.900220] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5426.075350] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5426.724860] Lustre: Permanently reactivating lustre-OST0000 [ 5426.732262] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5426.735785] Lustre: Skipped 2 previous similar messages [ 5451.743185] LustreError: 371498:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5455.070185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5455.178977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5455.181879] Lustre: Skipped 1 previous similar message [ 5457.345842] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5459.131288] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5471.714711] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5471.715045] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5471.720481] LustreError: Skipped 1 previous similar message [ 5471.721950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5471.729140] Lustre: Skipped 1 previous similar message [ 5473.394448] LustreError: 373274:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5473.398302] LustreError: 373274:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5473.462419] Lustre: server umount lustre-OST0000 complete [ 5476.833181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5476.839070] Lustre: Skipped 1 previous similar message [ 5477.167841] Lustre: server umount lustre-MDT0000 complete [ 5478.695613] LustreError: 369924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768145461 with bad export cookie 3085419043061521050 [ 5478.698891] LustreError: 369924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5478.700187] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5478.832196] Lustre: server umount lustre-MDT0001 complete [ 5482.773802] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 5483.653928] Key type lgssc unregistered [ 5483.772398] LNet: 374213:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5483.775394] LNetError: 374213:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5483.785382] LNet: Removed LNI 192.168.203.104@tcp [ 5484.070153] Key type .llcrypt unregistered [ 5484.071219] Key type ._llcrypt unregistered [ 5491.374408] Key type ._llcrypt registered [ 5491.375394] Key type .llcrypt registered [ 5491.413870] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 5498.179609] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5498.549538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5498.564093] alg: No test for adler32 (adler32-zlib) [ 5499.412839] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 5499.496053] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 5501.079171] Key type lgssc registered [ 5501.708591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5518.093584] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5518.103891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5519.206047] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5519.220396] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5519.266858] Lustre: lustre-MDT0000: new disk, initializing [ 5519.301457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5519.313260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5520.772761] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5525.562945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5525.621321] Lustre: 377306:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5525.656679] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5525.659821] Lustre: Skipped 1 previous similar message [ 5525.705914] Lustre: lustre-MDT0001: new disk, initializing [ 5525.743710] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5525.758617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5525.763410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5527.535175] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5531.087837] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5532.519763] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5535.668054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5535.815269] Lustre: lustre-OST0000: new disk, initializing [ 5535.818937] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5535.871251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5538.355122] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5542.033439] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5544.037073] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5544.974190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5544.980260] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5545.043801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5545.189803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5546.747715] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5546.854835] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5550.055499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5550.055675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5550.057889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5550.063663] Lustre: Skipped 1 previous similar message [ 5553.843179] LustreError: 380242:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5553.897578] Lustre: server umount lustre-OST0000 complete [ 5556.704692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5556.709453] LustreError: Skipped 1 previous similar message [ 5556.712169] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5556.719950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5556.724264] Lustre: Skipped 1 previous similar message [ 5560.289664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5560.290493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5561.880097] LustreError: 380443:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5561.883838] LustreError: 380443:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5561.920618] Lustre: server umount lustre-MDT0000 complete [ 5563.726332] LustreError: 377296:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768145546 with bad export cookie 15917540708484006583 [ 5563.732655] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5563.733840] LustreError: 377296:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5563.829357] LustreError: 380645:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5563.833157] LustreError: 380645:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5563.889289] Lustre: server umount lustre-MDT0001 complete [ 5567.503758] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5568.348385] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 10:32:30 (1768145550) [ 5571.396617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5571.648843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5573.361950] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5576.672482] LustreError: 381361:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5576.680027] LustreError: 381361:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 2 previous similar messages [ 5577.061150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5577.164688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5578.660186] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5580.958581] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5582.522122] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5585.789649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5585.942410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5588.411480] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5590.184202] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5591.088828] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5596.932395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5601.253146] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5601.253598] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5601.258215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5601.263667] Lustre: Skipped 1 previous similar message [ 5601.267093] Lustre: Skipped 1 previous similar message [ 5604.724162] LustreError: 383798:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5604.733486] LustreError: 383798:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5604.796988] Lustre: server umount lustre-OST0000 complete [ 5607.905034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5607.910072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5607.918791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5612.747924] LustreError: 384001:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5612.751014] LustreError: 384001:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5612.799626] Lustre: server umount lustre-MDT0000 complete [ 5614.553648] LustreError: 382078:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768145597 with bad export cookie 15917540708484007451 [ 5614.558760] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5614.559801] LustreError: 382078:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5614.701578] Lustre: server umount lustre-MDT0001 complete [ 5619.412867] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 5620.953279] Key type lgssc unregistered [ 5621.145622] LNet: 384734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5621.150086] LNetError: 384734:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5621.161636] LNet: Removed LNI 192.168.203.104@tcp [ 5621.645735] Key type .llcrypt unregistered [ 5621.647721] Key type ._llcrypt unregistered [ 5631.263312] Key type ._llcrypt registered [ 5631.265061] Key type .llcrypt registered [ 5631.308813] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 5631.406217] Key type .llcrypt unregistered [ 5631.407233] Key type ._llcrypt unregistered [ 5638.921621] Key type ._llcrypt registered [ 5638.922704] Key type .llcrypt registered [ 5638.961751] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5639.293333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5639.334604] alg: No test for adler32 (adler32-zlib) [ 5640.195185] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 5640.278711] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 5641.871181] Key type lgssc registered [ 5642.414759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5646.335412] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5646.346765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5647.536198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5649.244554] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5653.348732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5653.533083] LustreError: 387608:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5653.537464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5653.541759] LustreError: 387608:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 2 previous similar messages [ 5655.990298] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5657.637057] LustreError: 386881:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5657.645494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5657.645882] LustreError: 386881:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 1 previous similar message [ 5658.651243] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5662.064562] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5662.230267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5664.058920] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5672.420803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5672.421771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5672.427537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5672.432854] Lustre: Skipped 1 previous similar message [ 5677.542760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5677.546674] Lustre: Skipped 1 previous similar message [ 5678.255419] LustreError: 388934:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 5678.308320] Lustre: server umount lustre-OST0000 complete [ 5682.659371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5682.659407] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5682.672682] Lustre: Skipped 1 previous similar message [ 5682.676751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5682.680599] Lustre: Skipped 1 previous similar message [ 5686.159459] LustreError: 389137:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5686.163737] LustreError: 389137:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5686.200701] Lustre: server umount lustre-MDT0000 complete [ 5687.776954] LustreError: 386880:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5687.786639] LustreError: 386880:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 1 previous similar message [ 5688.100175] LustreError: 388498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768145671 with bad export cookie 4368147707020519890 [ 5688.106495] LustreError: 388498:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5688.108726] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5688.207174] LustreError: 389339:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5688.210299] LustreError: 389339:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5688.262067] Lustre: server umount lustre-MDT0001 complete [ 5693.144558] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 5694.626369] Key type lgssc unregistered [ 5694.798558] LNet: 389870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5694.803430] LNetError: 389870:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5694.811771] LNet: Removed LNI 192.168.203.104@tcp [ 5695.252329] Key type .llcrypt unregistered [ 5695.254248] Key type ._llcrypt unregistered [ 5706.188755] Key type ._llcrypt registered [ 5706.190220] Key type .llcrypt registered [ 5706.239619] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5706.620097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5706.627827] alg: No test for adler32 (adler32-zlib) [ 5707.517241] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 5707.635277] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 5709.239209] Key type lgssc registered [ 5709.884504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5713.947374] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5713.959945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5729.439434] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5734.559235] LustreError: 391219:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.104@tcp: failed processing log, type 4: rc = -110 [ 5761.183218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5763.583685] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5766.290409] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5769.728722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5769.988694] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5770.029161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5771.738587] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5775.140081] LustreError: 392765:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5775.151952] LustreError: 392765:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 3 previous similar messages [ 5775.862460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5776.043320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5777.809231] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5780.358737] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5782.003934] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5785.568620] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5785.574264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5785.583991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5786.595580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5786.595845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5786.599759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5790.964177] LustreError: 393493:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 5791.025589] Lustre: server umount lustre-OST0000 complete [ 5794.787509] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5794.792284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5794.800395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5794.805829] Lustre: Skipped 1 previous similar message [ 5794.897475] LustreError: 393695:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 5794.901095] LustreError: 393695:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5794.934508] Lustre: server umount lustre-MDT0000 complete [ 5796.762187] LustreError: 391227:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768145779 with bad export cookie 11618326011234781148 [ 5796.769041] LustreError: 391227:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5796.769884] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5796.861873] LustreError: 393897:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5796.864080] LustreError: 393897:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5796.919973] Lustre: server umount lustre-MDT0001 complete [ 5801.432456] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 5802.910694] Key type lgssc unregistered [ 5803.080712] LNet: 394428:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5803.087422] LNetError: 394428:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5803.096604] LNet: Removed LNI 192.168.203.104@tcp [ 5803.543580] Key type .llcrypt unregistered [ 5803.545739] Key type ._llcrypt unregistered [ 5815.599287] Key type ._llcrypt registered [ 5815.600629] Key type .llcrypt registered [ 5815.648191] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5815.981852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5816.021052] alg: No test for adler32 (adler32-zlib) [ 5816.873308] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 5816.953847] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 5818.527204] Key type lgssc registered [ 5819.086681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5822.950674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5822.961947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5824.148550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5825.363217] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5828.646103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5828.840336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5830.505566] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5832.918271] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5834.488480] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5837.803389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5837.970056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5840.414625] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5842.818929] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5850.286362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5854.688942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5854.693988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5854.703508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5855.716902] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5855.717816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5855.726480] Lustre: Skipped 1 previous similar message [ 5855.731869] Lustre: Skipped 1 previous similar message [ 5858.447812] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.4@tcp (stopping) [ 5860.751404] LustreError: 398063:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5860.786499] Lustre: server umount lustre-MDT0000 complete [ 5860.833993] LustreError: 395802:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5860.841190] LustreError: 395802:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 4 previous similar messages [ 5861.898694] LustreError: 395784:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.4@tcp arrived at 1768145844 with bad export cookie 16447929722443974841 [ 5861.908569] LustreError: 395784:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5862.867874] LustreError: 395784:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768145845 with bad export cookie 16447929722443974218 [ 5862.875913] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5862.875985] LustreError: 395784:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5864.928523] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5864.933527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5864.942454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5865.952434] LustreError: 395802:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5865.960079] LustreError: 395802:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 1 previous similar message [ 5868.000844] LustreError: 395802:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5868.011759] LustreError: 395802:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 1 previous similar message [ 5868.979134] LustreError: 398267:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5868.983292] LustreError: 398267:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5869.034714] Lustre: server umount lustre-OST0000 complete [ 5872.098085] LustreError: 395802:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5872.106969] LustreError: 395802:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 1 previous similar message [ 5872.518155] LustreError: 398622:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5872.521033] LustreError: 398622:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5872.572277] Lustre: server umount lustre-MDT0001 complete [ 5877.511166] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 5878.936349] Key type lgssc unregistered [ 5879.098665] LNet: 399153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5879.103729] LNetError: 399153:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5879.113499] LNet: Removed LNI 192.168.203.104@tcp [ 5879.560986] Key type .llcrypt unregistered [ 5879.562943] Key type ._llcrypt unregistered [ 5891.448190] Key type ._llcrypt registered [ 5891.449251] Key type .llcrypt registered [ 5891.497014] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5892.033791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5892.045368] alg: No test for adler32 (adler32-zlib) [ 5892.938151] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 5893.066561] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 5894.679212] Key type lgssc registered [ 5895.326831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5899.116262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5899.125353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5900.367225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5902.151353] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5906.603346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5906.788131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5908.209558] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5910.574101] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5912.367646] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5915.715914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5915.883738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5918.419970] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 5921.063202] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5926.124629] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5931.493232] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5931.496435] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5931.496860] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5931.502675] Lustre: Skipped 1 previous similar message [ 5933.773562] LustreError: 402769:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5933.806185] Lustre: server umount lustre-MDT0001 complete [ 5936.609487] LustreError: 401239:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5936.609859] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5936.610723] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5936.611967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5936.611974] Lustre: Skipped 2 previous similar messages [ 5936.618824] LustreError: 401239:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 2 previous similar messages [ 5941.729398] LustreError: 400527:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5941.730940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5942.258183] LustreError: 402974:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5942.262141] LustreError: 402974:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5942.306481] Lustre: server umount lustre-OST0000 complete [ 5944.050869] LustreError: 403175:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5944.053238] LustreError: 403175:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5944.104317] Lustre: server umount lustre-MDT0000 complete [ 5949.538844] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 5950.987569] Key type lgssc unregistered [ 5951.158674] LNet: 403858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5951.163413] LNetError: 403858:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5951.174542] LNet: Removed LNI 192.168.203.104@tcp [ 5951.617780] Key type .llcrypt unregistered [ 5951.619962] Key type ._llcrypt unregistered [ 5962.820417] Key type ._llcrypt registered [ 5962.821872] Key type .llcrypt registered [ 5962.883826] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 5970.167222] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5970.701370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5970.709667] alg: No test for adler32 (adler32-zlib) [ 5971.590376] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 5971.705372] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 5973.319172] Key type lgssc registered [ 5973.970210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5991.250257] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 5998.576806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5998.587137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5999.708562] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5999.720915] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5999.759516] Lustre: lustre-MDT0000: new disk, initializing [ 5999.793898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5999.802440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6001.542120] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6009.369574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6009.555875] Lustre: lustre-OST0000: new disk, initializing [ 6009.559707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6009.563279] Lustre: Skipped 1 previous similar message [ 6009.606816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6012.197564] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6013.290910] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6013.296266] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6013.312087] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6016.230613] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6023.464249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6023.534870] Lustre: 407791:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6023.573917] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6023.622855] Lustre: lustre-MDT0001: new disk, initializing [ 6023.670846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6023.686923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6023.690341] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6025.275456] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6029.480766] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6029.601725] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6031.325776] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6032.483880] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [ 6035.602052] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.4@tcp (stopping) [ 6036.450104] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6036.453407] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6036.455957] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6036.461909] Lustre: Skipped 1 previous similar message [ 6036.470886] Lustre: Skipped 1 previous similar message [ 6040.706301] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.4@tcp (stopping) [ 6040.854471] LustreError: 411207:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6040.883842] Lustre: server umount lustre-MDT0001 complete [ 6046.691474] LustreError: 407797:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6046.700813] LustreError: 407797:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 4 previous similar messages [ 6051.811315] LustreError: 407798:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6051.819832] LustreError: 407798:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 1 previous similar message [ 6054.368441] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6054.375553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6054.386224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6056.931389] LustreError: 407803:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6062.049303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6062.050621] LustreError: 407803:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6062.053160] Lustre: Skipped 1 previous similar message [ 6067.170906] LustreError: 407803:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6068.191136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6068.220166] LustreError: 411481:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 6068.225305] LustreError: 411481:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6068.278793] Lustre: server umount lustre-OST0000 complete [ 6076.235734] LustreError: 411683:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6076.240431] LustreError: 411683:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6076.309643] Lustre: server umount lustre-MDT0000 complete [ 6082.105091] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 6083.439402] Key type lgssc unregistered [ 6083.595599] LNet: 412368:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6083.600184] LNetError: 412368:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6083.610572] LNet: Removed LNI 192.168.203.104@tcp [ 6084.036776] Key type .llcrypt unregistered [ 6084.038960] Key type ._llcrypt unregistered [ 6099.837825] Key type ._llcrypt registered [ 6099.839166] Key type .llcrypt registered [ 6099.884748] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 6100.020308] Key type .llcrypt unregistered [ 6100.022305] Key type ._llcrypt unregistered [ 6107.621961] Key type ._llcrypt registered [ 6107.623287] Key type .llcrypt registered [ 6107.667182] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6107.987348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6108.048761] alg: No test for adler32 (adler32-zlib) [ 6108.898443] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 6108.982922] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 6110.559201] Key type lgssc registered [ 6111.078592] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6122.141256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6122.150093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6123.245173] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6123.254795] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6123.290195] Lustre: lustre-MDT0000: new disk, initializing [ 6123.311506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6123.317229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6124.918499] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6129.737047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6129.795714] Lustre: 415684:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6129.837804] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6129.841021] Lustre: Skipped 1 previous similar message [ 6129.890635] Lustre: lustre-MDT0001: new disk, initializing [ 6129.944996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6129.965946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6129.972409] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6131.262521] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6134.536467] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6136.067058] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6139.128166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6139.283922] Lustre: lustre-OST0000: new disk, initializing [ 6139.286776] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6139.335600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6140.986734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6140.991491] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6141.038681] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6141.508647] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6144.725220] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6150.624633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6150.631065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6150.640467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6151.140102] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6151.141223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6151.147710] Lustre: Skipped 1 previous similar message [ 6151.151549] Lustre: Skipped 1 previous similar message [ 6153.147537] LustreError: 418120:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6153.177490] Lustre: server umount lustre-MDT0000 complete [ 6154.746385] LustreError: 415675:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768146137 with bad export cookie 4038419230140851612 [ 6154.752705] LustreError: 415675:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6154.754646] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6154.832270] LustreError: 418321:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6154.835190] LustreError: 418321:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6154.888461] Lustre: server umount lustre-MDT0001 complete [ 6166.520138] LustreError: 418521:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6166.523988] LustreError: 418521:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6166.602495] Lustre: server umount lustre-OST0000 complete [ 6170.544061] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 10:42:32 (1768146152) [ 6173.379271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6173.606270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6175.203463] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6178.667782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6178.791682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6179.976680] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6182.303565] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6183.568858] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6197.768045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6197.899541] Lustre: Found index 0 for lustre-OST0000, updating log [ 6197.994648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6199.020676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6200.502894] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6202.631043] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6204.384207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6204.386558] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6204.390469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6208.481873] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6208.483844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6208.487157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6210.092739] LustreError: 421584:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6210.094704] LustreError: 421584:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6210.127889] Lustre: server umount lustre-OST0000 complete [ 6224.864572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6224.871278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6224.883993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6224.889092] Lustre: Skipped 2 previous similar messages [ 6229.436511] LustreError: 421846:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6229.438263] LustreError: 421846:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6229.465164] Lustre: server umount lustre-MDT0000 complete [ 6230.927671] LustreError: 419319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768146213 with bad export cookie 4038419230140852991 [ 6230.931972] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6230.933374] LustreError: 419319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6231.080677] Lustre: server umount lustre-MDT0001 complete [ 6234.779122] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 10:43:36 (1768146216) [ 6256.479140] LNet: 1 local NIs in recovery (showing 1): 192.168.203.104@tcp [ 6819.678310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6819.906107] LustreError: 425407:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6819.917095] LustreError: 425407:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 2 previous similar messages [ 6819.946627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6821.619118] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6825.440976] LustreError: 425407:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6825.682439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6827.496954] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6830.125750] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6831.925820] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6846.140426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6846.267861] Lustre: Found index 0 for lustre-OST0000, updating log [ 6846.310482] LustreError: 427054:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6848.893687] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 6851.364672] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6852.382161] LustreError: 427700:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6852.386301] LustreError: 427700:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6852.439309] Lustre: server umount lustre-OST0000 complete [ 6856.672314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6856.676686] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6856.683416] Lustre: Skipped 1 previous similar message [ 6856.686460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6856.689802] Lustre: Skipped 2 previous similar messages [ 6858.719930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6858.720507] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6860.368444] LustreError: 427908:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6860.372458] LustreError: 427908:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6860.409184] Lustre: server umount lustre-MDT0000 complete [ 6862.191139] LustreError: 427081:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768146845 with bad export cookie 4038419230140854069 [ 6862.197929] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6862.198312] LustreError: 427081:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6862.348528] Lustre: server umount lustre-MDT0001 complete [ 6870.879178] LNet: 1 local NIs in recovery (showing 1): 192.168.203.104@tcp [ 6881.955271] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 6886.527412] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6904.578433] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 10:54:46 (1768146886) [ 6912.881888] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 6914.296605] Key type lgssc unregistered [ 6914.482962] LNet: 431658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6914.487607] LNetError: 431658:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6976.997566] LNet: Removed LNI 192.168.203.104@tcp [ 6977.458057] Key type .llcrypt unregistered [ 6977.460394] Key type ._llcrypt unregistered [ 6986.156396] Key type ._llcrypt registered [ 6986.157946] Key type .llcrypt registered [ 6986.221063] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 6986.550578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6986.581090] alg: No test for adler32 (adler32-zlib) [ 6987.461907] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 6987.569685] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 6989.159136] Key type lgssc registered [ 6989.718265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6991.814238] LNet: Added LNI 192.168.203.104@tcp100 [8/256/0/180] [ 6992.201436] LNet: Added LNI 192.168.203.104@tcp101 [8/256/0/180] [ 6992.627923] LNet: Added LNI 192.168.203.104@tcp102 [8/256/0/180] [ 6993.043259] LNet: Added LNI 192.168.203.104@tcp103 [8/256/0/180] [ 6993.467819] LNet: Added LNI 192.168.203.104@tcp104 [8/256/0/180] [ 6993.900776] LNet: Added LNI 192.168.203.104@tcp105 [8/256/0/180] [ 6994.302680] LNet: Added LNI 192.168.203.104@tcp106 [8/256/0/180] [ 6994.725640] LNet: Added LNI 192.168.203.104@tcp107 [8/256/0/180] [ 6995.154293] LNet: Added LNI 192.168.203.104@tcp108 [8/256/0/180] [ 6995.582935] LNet: Added LNI 192.168.203.104@tcp109 [8/256/0/180] [ 6996.008892] LNet: Added LNI 192.168.203.104@tcp110 [8/256/0/180] [ 6996.426816] LNet: Added LNI 192.168.203.104@tcp111 [8/256/0/180] [ 6996.856446] LNet: Added LNI 192.168.203.104@tcp112 [8/256/0/180] [ 6997.271112] LNet: Added LNI 192.168.203.104@tcp113 [8/256/0/180] [ 6997.689576] LNet: Added LNI 192.168.203.104@tcp114 [8/256/0/180] [ 6998.128725] LNet: Added LNI 192.168.203.104@tcp115 [8/256/0/180] [ 6998.565831] LNet: Added LNI 192.168.203.104@tcp116 [8/256/0/180] [ 6998.995333] LNet: Added LNI 192.168.203.104@tcp117 [8/256/0/180] [ 6999.426590] LNet: Added LNI 192.168.203.104@tcp118 [8/256/0/180] [ 6999.813918] LNet: Added LNI 192.168.203.104@tcp119 [8/256/0/180] [ 7000.156721] LNet: Added LNI 192.168.203.104@tcp120 [8/256/0/180] [ 7000.560865] LNet: Added LNI 192.168.203.104@tcp121 [8/256/0/180] [ 7000.960366] LNet: Added LNI 192.168.203.104@tcp122 [8/256/0/180] [ 7001.354466] LNet: Added LNI 192.168.203.104@tcp123 [8/256/0/180] [ 7001.760594] LNet: Added LNI 192.168.203.104@tcp124 [8/256/0/180] [ 7002.108922] LNet: Added LNI 192.168.203.104@tcp125 [8/256/0/180] [ 7002.465420] LNet: Added LNI 192.168.203.104@tcp126 [8/256/0/180] [ 7002.772947] LNet: Added LNI 192.168.203.104@tcp127 [8/256/0/180] [ 7003.066425] LNet: Added LNI 192.168.203.104@tcp128 [8/256/0/180] [ 7003.328835] LNet: Added LNI 192.168.203.104@tcp129 [8/256/0/180] [ 7003.589524] LNet: Added LNI 192.168.203.104@tcp130 [8/256/0/180] [ 7003.857474] LNet: Added LNI 192.168.203.104@tcp131 [8/256/0/180] [ 7004.191223] LNet: Added LNI 192.168.203.104@tcp132 [8/256/0/180] [ 7004.581482] LNet: Added LNI 192.168.203.104@tcp133 [8/256/0/180] [ 7004.983853] LNet: Added LNI 192.168.203.104@tcp134 [8/256/0/180] [ 7005.402578] LNet: Added LNI 192.168.203.104@tcp135 [8/256/0/180] [ 7008.323659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7008.333451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7009.389379] Lustre: 434637:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.104@tcp: can't add NID 192.168.203.104@tcp130: rc = -75 [ 7009.416563] LNetError: 434658:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.104 [ 7009.419839] LNetError: Protocol error connecting to 192.168.203.104@tcp116 at host 192.168.203.104:988: is it running a compatible version of Lustre? [ 7009.468942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7009.482449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7009.518339] Lustre: lustre-MDT0000: new disk, initializing [ 7009.549966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7009.557398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7010.468384] LNetError: 434659:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.104 [ 7010.474541] LNetError: Protocol error connecting to 192.168.203.104@tcp106 at host 192.168.203.104:988: is it running a compatible version of Lustre? [ 7011.201763] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7016.594058] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7016.636872] Lustre: 434654:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7016.660196] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7016.661699] Lustre: Skipped 1 previous similar message [ 7016.695255] Lustre: lustre-MDT0001: new disk, initializing [ 7016.705647] Lustre: 435438:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.104@tcp: can't add NID 192.168.203.104@tcp130: rc = -75 [ 7016.708064] Lustre: 435438:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7016.742275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7016.752561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7016.755242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7018.372549] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7022.161305] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7023.764550] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7026.933026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7027.038529] Lustre: lustre-OST0000: new disk, initializing [ 7027.041395] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7027.076705] Lustre: 436404:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.104@tcp: can't add NID 192.168.203.104@tcp130: rc = -75 [ 7027.080415] Lustre: 436404:0:(lustre_peer.c:105:class_add_uuid()) Skipped 17 previous similar messages [ 7027.098669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7029.389160] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7032.305472] Lustre: 436950:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.104@tcp: can't add NID 192.168.203.104@tcp130: rc = -75 [ 7032.307946] Lustre: 436950:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7032.320944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7032.323594] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7032.360435] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7033.042503] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7037.412435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7037.413553] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7037.415889] LustreError: Skipped 1 previous similar message [ 7037.421761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7037.425850] Lustre: Skipped 1 previous similar message [ 7038.513204] LustreError: 437237:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7038.573719] Lustre: server umount lustre-OST0000 complete [ 7042.529866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7042.530730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7042.542098] Lustre: Skipped 1 previous similar message [ 7042.545983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7042.550179] Lustre: Skipped 1 previous similar message [ 7043.872756] LNetError: 432100:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.104 [ 7043.879206] LNetError: Protocol error connecting to 192.168.203.104@tcp116 at host 192.168.203.104:988: is it running a compatible version of Lustre? [ 7045.920583] LNetError: 432098:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.104 [ 7045.926913] LNetError: Protocol error connecting to 192.168.203.104@tcp106 at host 192.168.203.104:988: is it running a compatible version of Lustre? [ 7046.478948] LustreError: 437440:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7046.483137] LustreError: 437440:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7046.517166] Lustre: server umount lustre-MDT0000 complete [ 7048.302638] LustreError: 434646:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768147031 with bad export cookie 7975740238820135275 [ 7048.308277] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7048.309753] LustreError: 434646:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7048.410548] LustreError: 437643:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7048.413842] LustreError: 437643:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7048.470367] Lustre: server umount lustre-MDT0001 complete [ 7053.233685] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7054.721437] Key type lgssc unregistered [ 7054.933555] LNet: 438182:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7054.938787] LNetError: 438182:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7054.943282] LNet: Removed LNI 192.168.203.104@tcp [ 7055.967158] LNet: Removed LNI 192.168.203.104@tcp100 [ 7055.969860] LNetError: 438182:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7055.974488] LNetError: 438182:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7055.979501] LNet: Removed LNI 192.168.203.104@tcp101 [ 7056.991186] LNet: Removed LNI 192.168.203.104@tcp102 [ 7056.994166] LNetError: 438182:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7056.998815] LNetError: 438182:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7057.003759] LNet: Removed LNI 192.168.203.104@tcp103 [ 7058.015156] LNet: Removed LNI 192.168.203.104@tcp104 [ 7059.039158] LNet: Removed LNI 192.168.203.104@tcp105 [ 7059.042184] LNetError: 438182:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7059.047014] LNetError: 438182:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7059.052206] LNet: Removed LNI 192.168.203.104@tcp106 [ 7060.063167] LNet: Removed LNI 192.168.203.104@tcp107 [ 7061.087180] LNet: Removed LNI 192.168.203.104@tcp108 [ 7062.111148] LNet: Removed LNI 192.168.203.104@tcp109 [ 7063.135190] LNet: Removed LNI 192.168.203.104@tcp110 [ 7063.137123] LNetError: 438182:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7063.139827] LNetError: 438182:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7063.142898] LNet: Removed LNI 192.168.203.104@tcp111 [ 7064.159184] LNet: Removed LNI 192.168.203.104@tcp112 [ 7065.183199] LNet: Removed LNI 192.168.203.104@tcp113 [ 7066.207209] LNet: Removed LNI 192.168.203.104@tcp114 [ 7067.231169] LNet: Removed LNI 192.168.203.104@tcp115 [ 7068.255170] LNet: Removed LNI 192.168.203.104@tcp116 [ 7069.279162] LNet: Removed LNI 192.168.203.104@tcp117 [ 7070.303165] LNet: Removed LNI 192.168.203.104@tcp118 [ 7070.305830] LNet: Removed LNI 192.168.203.104@tcp119 [ 7070.308310] LNet: Removed LNI 192.168.203.104@tcp120 [ 7070.310766] LNet: Removed LNI 192.168.203.104@tcp121 [ 7070.313210] LNet: Removed LNI 192.168.203.104@tcp122 [ 7070.315728] LNet: Removed LNI 192.168.203.104@tcp123 [ 7070.318197] LNet: Removed LNI 192.168.203.104@tcp124 [ 7070.320750] LNet: Removed LNI 192.168.203.104@tcp125 [ 7070.323220] LNet: Removed LNI 192.168.203.104@tcp126 [ 7070.325751] LNet: Removed LNI 192.168.203.104@tcp127 [ 7070.328215] LNet: Removed LNI 192.168.203.104@tcp128 [ 7070.330792] LNet: Removed LNI 192.168.203.104@tcp129 [ 7070.333255] LNet: Removed LNI 192.168.203.104@tcp130 [ 7070.335737] LNet: Removed LNI 192.168.203.104@tcp131 [ 7070.338165] LNet: Removed LNI 192.168.203.104@tcp132 [ 7070.340730] LNet: Removed LNI 192.168.203.104@tcp133 [ 7070.343179] LNet: Removed LNI 192.168.203.104@tcp134 [ 7070.352035] LNet: Removed LNI 192.168.203.104@tcp135 [ 7070.763982] Key type .llcrypt unregistered [ 7070.766130] Key type ._llcrypt unregistered [ 7081.393760] Key type ._llcrypt registered [ 7081.394849] Key type .llcrypt registered [ 7081.428280] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 7089.085323] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7089.573758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7089.600853] alg: No test for adler32 (adler32-zlib) [ 7090.485847] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 7090.596764] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 7092.183173] Key type lgssc registered [ 7092.620807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7108.307128] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 10:58:10 (1768147090) [ 7115.535117] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7116.627624] Key type lgssc unregistered [ 7116.809807] LNet: 442185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7116.814397] LNetError: 442185:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7116.826645] LNet: Removed LNI 192.168.203.104@tcp [ 7117.248680] Key type .llcrypt unregistered [ 7117.250757] Key type ._llcrypt unregistered [ 7124.732657] Key type ._llcrypt registered [ 7124.734017] Key type .llcrypt registered [ 7124.801394] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7125.179552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7125.187445] alg: No test for adler32 (adler32-zlib) [ 7126.071952] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 7126.177338] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 7127.783181] Key type lgssc registered [ 7128.373091] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7132.881154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7132.890913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7134.019604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7134.034578] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7134.084599] Lustre: lustre-MDT0000: new disk, initializing [ 7134.120789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7134.129160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7135.920852] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7141.490247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7141.552691] Lustre: 443355:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7141.605488] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7141.610610] Lustre: Skipped 1 previous similar message [ 7141.664165] Lustre: lustre-MDT0001: new disk, initializing [ 7141.719850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7141.737590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7141.744274] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7143.627829] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7147.513234] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7149.279367] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7152.623287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7152.746360] Lustre: lustre-OST0000: new disk, initializing [ 7152.748642] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7152.784768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7155.476081] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7159.334727] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7160.932736] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing lnet_if_list [ 7162.739709] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7162.751646] LNet: Added LNI 192.168.203.104@tcp10 [8/256/0/180] [ 7162.891901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7162.896533] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7162.942202] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7165.028990] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7165.049290] LNet: Added LNI 192.168.203.104@tcp11 [8/256/0/180] [ 7166.920888] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7166.935661] LNet: Added LNI 192.168.203.104@tcp12 [8/256/0/180] [ 7168.843779] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7168.860677] LNet: Added LNI 192.168.203.104@tcp13 [8/256/0/180] [ 7170.974404] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7170.995125] LNet: Added LNI 192.168.203.104@tcp14 [8/256/0/180] [ 7172.958562] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7172.972422] LNet: Added LNI 192.168.203.104@tcp15 [8/256/0/180] [ 7174.638072] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7174.648709] LNet: Added LNI 192.168.203.104@tcp16 [8/256/0/180] [ 7176.277165] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7176.290966] LNet: Added LNI 192.168.203.104@tcp17 [8/256/0/180] [ 7178.205777] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7178.222906] LNet: Added LNI 192.168.203.104@tcp18 [8/256/0/180] [ 7180.569275] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7180.589327] LNet: Added LNI 192.168.203.104@tcp19 [8/256/0/180] [ 7183.066284] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7183.086310] LNet: Added LNI 192.168.203.104@tcp20 [8/256/0/180] [ 7185.602244] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7185.617042] LNet: Added LNI 192.168.203.104@tcp21 [8/256/0/180] [ 7187.902772] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7187.921824] LNet: Added LNI 192.168.203.104@tcp22 [8/256/0/180] [ 7190.315167] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7190.330940] LNet: Added LNI 192.168.203.104@tcp23 [8/256/0/180] [ 7192.515336] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7192.533641] LNet: Added LNI 192.168.203.104@tcp24 [8/256/0/180] [ 7194.859921] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7194.872820] LNet: Added LNI 192.168.203.104@tcp25 [8/256/0/180] [ 7197.089772] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7197.102391] LNet: Added LNI 192.168.203.104@tcp26 [8/256/0/180] [ 7199.434411] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7199.452814] LNet: Added LNI 192.168.203.104@tcp27 [8/256/0/180] [ 7199.463125] LNetError: 442619:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.104 [ 7199.469943] LNetError: Protocol error connecting to 192.168.203.104@tcp26 at host 192.168.203.104:988: is it running a compatible version of Lustre? [ 7201.966340] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7201.985237] LNet: Added LNI 192.168.203.104@tcp28 [8/256/0/180] [ 7201.991878] LNetError: 442618:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.104 [ 7201.998756] LNetError: Protocol error connecting to 192.168.203.104@tcp26 at host 192.168.203.104:988: is it running a compatible version of Lustre? [ 7204.370766] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7204.388155] LNet: Added LNI 192.168.203.104@tcp29 [8/256/0/180] [ 7204.395498] LNetError: 442617:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.104 [ 7204.404801] LNetError: Protocol error connecting to 192.168.203.104@tcp28 at host 192.168.203.104:988: is it running a compatible version of Lustre? [ 7205.023592] LNet: 1 local NIs in recovery (showing 1): 192.168.203.104@tcp28 [ 7207.073379] LNetError: 442617:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.104 [ 7207.075702] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7207.078435] LNetError: 442617:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 3 previous similar messages [ 7207.088553] LNetError: Protocol error connecting to 192.168.203.104@tcp28 at host 192.168.203.104:988: is it running a compatible version of Lustre? [ 7207.093485] LNet: Added LNI 192.168.203.104@tcp30 [8/256/0/180] [ 7207.094164] LNetError: Skipped 3 previous similar messages [ 7209.561418] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7209.574105] LNet: Added LNI 192.168.203.104@tcp31 [8/256/0/180] [ 7211.168744] LNetError: 442617:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.104 [ 7211.174661] LNetError: 442617:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 3 previous similar messages [ 7211.179454] LNetError: Protocol error connecting to 192.168.203.104@tcp28 at host 192.168.203.104:988: is it running a compatible version of Lustre? [ 7211.185624] LNetError: Skipped 3 previous similar messages [ 7212.087180] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7212.104914] LNet: Added LNI 192.168.203.104@tcp32 [8/256/0/180] [ 7214.776408] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7214.789817] LNet: Added LNI 192.168.203.104@tcp33 [8/256/0/180] [ 7217.393228] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7217.407337] LNet: Added LNI 192.168.203.104@tcp34 [8/256/0/180] [ 7219.361703] LNetError: 442619:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.104 [ 7219.369493] LNetError: 442619:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 8 previous similar messages [ 7219.374727] LNetError: Protocol error connecting to 192.168.203.104@tcp28 at host 192.168.203.104:988: is it running a compatible version of Lustre? [ 7219.381532] LNetError: Skipped 8 previous similar messages [ 7220.171272] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7220.190262] LNet: Added LNI 192.168.203.104@tcp35 [8/256/0/180] [ 7223.303830] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7223.322323] LNet: Added LNI 192.168.203.104@tcp36 [8/256/0/180] [ 7226.342948] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7226.354170] LNet: Added LNI 192.168.203.104@tcp37 [8/256/0/180] [ 7226.990309] LNetError: 442626:0:(peer.c:1828:lnet_peer_add_nid()) Peer 192.168.203.104@tcp NID 192.168.203.104@tcp is already locked with peer 192.168.203.104@tcp [ 7226.997369] LNetError: 442626:0:(lib-move.c:2321:lnet_handle_find_routed_path()) peer 192.168.203.104@tcp has no available nets [ 7229.279783] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7229.296955] LNet: Added LNI 192.168.203.104@tcp38 [8/256/0/180] [ 7229.300251] LNetError: 442626:0:(lib-move.c:2321:lnet_handle_find_routed_path()) peer 192.168.203.104@tcp has no available nets [ 7232.361493] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7232.378917] LNet: Added LNI 192.168.203.104@tcp39 [8/256/0/180] [ 7232.382252] LNetError: 442626:0:(lib-move.c:2321:lnet_handle_find_routed_path()) peer 192.168.203.104@tcp has no available nets [ 7232.387498] LNetError: 442626:0:(lib-move.c:2321:lnet_handle_find_routed_path()) Skipped 1 previous similar message [ 7235.700589] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7235.720061] LNet: Added LNI 192.168.203.104@tcp40 [8/256/0/180] [ 7235.752763] LNetError: 442618:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.104 [ 7235.756510] LNetError: 442618:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 29 previous similar messages [ 7235.759721] LNetError: Protocol error connecting to 192.168.203.104@tcp38 at host 192.168.203.104:988: is it running a compatible version of Lustre? [ 7235.763403] LNetError: Skipped 30 previous similar messages [ 7239.436635] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7239.460203] LNet: Added LNI 192.168.203.104@tcp41 [8/256/0/180] [ 7243.236533] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7243.257049] LNet: Added LNI 192.168.203.104@tcp42 [8/256/0/180] [ 7247.208554] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7247.228340] LNet: Added LNI 192.168.203.104@tcp43 [8/256/0/180] [ 7251.247400] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7251.267214] LNet: Added LNI 192.168.203.104@tcp44 [8/256/0/180] [ 7255.223303] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7255.241049] LNet: Added LNI 192.168.203.104@tcp45 [8/256/0/180] [ 7258.781291] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7258.801520] LNet: Added LNI 192.168.203.104@tcp46 [8/256/0/180] [ 7262.598827] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7262.613432] LNet: Added LNI 192.168.203.104@tcp47 [8/256/0/180] [ 7264.415280] LNet: 9 local NIs in recovery (showing 9): 192.168.203.104@tcp45, 192.168.203.104@tcp43, 192.168.203.104@tcp44, 192.168.203.104@tcp41, 192.168.203.104@tcp40, 192.168.203.104@tcp38, 192.168.203.104@tcp32, 192.168.203.104@tcp31, 192.168.203.104@tcp30 [ 7266.480278] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7266.500909] LNet: Added LNI 192.168.203.104@tcp48 [8/256/0/180] [ 7268.513436] LNetError: 442618:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.104 [ 7268.521565] LNetError: 442618:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 41 previous similar messages [ 7268.527667] LNetError: Protocol error connecting to 192.168.203.104@tcp37 at host 192.168.203.104:988: is it running a compatible version of Lustre? [ 7268.535541] LNetError: Skipped 40 previous similar messages [ 7270.685401] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7270.704962] LNet: Added LNI 192.168.203.104@tcp49 [8/256/0/180] [ 7274.528281] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7274.543711] LNet: Added LNI 192.168.203.104@tcp50 [8/256/0/180] [ 7285.732676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7285.734546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7285.736481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7285.737289] LustreError: Skipped 1 previous similar message [ 7285.744831] Lustre: Skipped 1 previous similar message [ 7290.675198] LustreError: 467563:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7290.734542] Lustre: server umount lustre-OST0000 complete [ 7295.456381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7295.461162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7295.469462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7295.473212] Lustre: Skipped 1 previous similar message [ 7298.710742] LustreError: 467764:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7298.714114] LustreError: 467764:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7298.744103] Lustre: server umount lustre-MDT0000 complete [ 7300.488062] LustreError: 443348:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768147283 with bad export cookie 15506095692494209406 [ 7300.494424] LustreError: 443348:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7300.494902] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7300.595865] LustreError: 467966:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7300.598929] LustreError: 467966:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7300.656461] Lustre: server umount lustre-MDT0001 complete [ 7305.101757] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7306.562466] Key type lgssc unregistered [ 7306.778360] LNet: 468505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7306.786579] LNetError: 468505:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7306.793429] LNet: Removed LNI 192.168.203.104@tcp [ 7307.807203] LNet: Removed LNI 192.168.203.104@tcp10 [ 7307.811709] LNetError: 468505:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7307.818726] LNetError: 468505:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7307.826849] LNet: Removed LNI 192.168.203.104@tcp11 [ 7308.831246] LNet: Removed LNI 192.168.203.104@tcp12 [ 7308.835132] LNetError: 468505:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7308.842016] LNetError: 468505:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7308.847375] LNet: Removed LNI 192.168.203.104@tcp13 [ 7309.855221] LNet: Removed LNI 192.168.203.104@tcp14 [ 7310.879160] LNet: Removed LNI 192.168.203.104@tcp15 [ 7310.882146] LNetError: 468505:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7310.887179] LNetError: 468505:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7310.892434] LNet: Removed LNI 192.168.203.104@tcp16 [ 7311.903134] LNet: Removed LNI 192.168.203.104@tcp17 [ 7312.927176] LNet: Removed LNI 192.168.203.104@tcp18 [ 7313.951218] LNet: Removed LNI 192.168.203.104@tcp19 [ 7314.975252] LNet: Removed LNI 192.168.203.104@tcp20 [ 7314.977947] LNetError: 468505:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7314.982384] LNetError: 468505:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7314.987321] LNet: Removed LNI 192.168.203.104@tcp21 [ 7315.999219] LNet: Removed LNI 192.168.203.104@tcp22 [ 7317.023232] LNet: Removed LNI 192.168.203.104@tcp23 [ 7318.047167] LNet: Removed LNI 192.168.203.104@tcp24 [ 7319.071163] LNet: Removed LNI 192.168.203.104@tcp25 [ 7320.095176] LNet: Removed LNI 192.168.203.104@tcp26 [ 7321.119213] LNet: Removed LNI 192.168.203.104@tcp27 [ 7322.143131] LNet: Removed LNI 192.168.203.104@tcp28 [ 7323.167195] LNet: Removed LNI 192.168.203.104@tcp29 [ 7323.169787] LNetError: 468505:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7323.174132] LNetError: 468505:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 7323.178913] LNet: Removed LNI 192.168.203.104@tcp30 [ 7324.191207] LNet: Removed LNI 192.168.203.104@tcp31 [ 7324.193875] LNet: Removed LNI 192.168.203.104@tcp32 [ 7325.215178] LNet: Removed LNI 192.168.203.104@tcp33 [ 7326.239245] LNet: Removed LNI 192.168.203.104@tcp34 [ 7327.263197] LNet: Removed LNI 192.168.203.104@tcp35 [ 7328.287208] LNet: Removed LNI 192.168.203.104@tcp36 [ 7329.311202] LNet: Removed LNI 192.168.203.104@tcp37 [ 7330.335165] LNet: Removed LNI 192.168.203.104@tcp38 [ 7331.359168] LNet: Removed LNI 192.168.203.104@tcp39 [ 7331.361750] LNet: Removed LNI 192.168.203.104@tcp40 [ 7331.364177] LNet: Removed LNI 192.168.203.104@tcp41 [ 7332.383180] LNet: Removed LNI 192.168.203.104@tcp42 [ 7332.385768] LNet: Removed LNI 192.168.203.104@tcp43 [ 7333.407170] LNet: Removed LNI 192.168.203.104@tcp44 [ 7333.409237] LNet: Removed LNI 192.168.203.104@tcp45 [ 7333.411096] LNet: Removed LNI 192.168.203.104@tcp46 [ 7334.431199] LNet: Removed LNI 192.168.203.104@tcp47 [ 7335.455200] LNet: Removed LNI 192.168.203.104@tcp48 [ 7336.479167] LNet: Removed LNI 192.168.203.104@tcp49 [ 7336.487908] LNet: Removed LNI 192.168.203.104@tcp50 [ 7336.870256] LNetError: 468568:0:(module.c:692:libcfs_exit()) Portals memory leaked: 752 bytes [ 7336.898512] Key type .llcrypt unregistered [ 7336.900654] Key type ._llcrypt unregistered [ 7346.904376] Key type ._llcrypt registered [ 7346.905942] Key type .llcrypt registered [ 7346.964323] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7347.394064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7347.403477] alg: No test for adler32 (adler32-zlib) [ 7348.311883] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 7348.437212] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 7350.015157] Key type lgssc registered [ 7350.568572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7354.198829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7354.209979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7355.456922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7357.310733] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7361.250640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7361.431020] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7362.801496] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7365.396212] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7367.259940] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7370.572983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7370.732933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7373.349311] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7376.030279] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7386.343024] loop0: detected capacity change from 0 to 204800000 [ 7386.397799] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7386.402835] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7386.407631] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7389.405911] loop0: detected capacity change from 0 to 204800000 [ 7389.481864] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7389.489715] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7389.494634] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7391.993301] loop0: detected capacity change from 0 to 204800000 [ 7394.527516] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7394.588243] Lustre: fs15246-OST0000: new disk, initializing [ 7394.591437] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7394.619126] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7397.254281] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7402.383389] loop1: detected capacity change from 0 to 204800000 [ 7405.029905] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7405.092973] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7405.141994] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7405.192198] Lustre: fs15246-MDT0000: new disk, initializing [ 7405.289430] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7405.303943] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7405.372459] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7406.432220] Lustre: 474093:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000fe5f9924 x1854036855784192/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1768147405 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7407.158669] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7415.525315] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7415.531080] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7415.545419] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7418.349975] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7418.358651] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7423.464905] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7423.506727] LustreError: 475216:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 7423.543398] Lustre: server umount fs15246-MDT0000 complete [ 7428.073144] LustreError: 475719:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 7428.077214] LustreError: 475719:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7428.104323] Lustre: server umount fs15246-OST0000 complete [ 7433.185762] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7433.190405] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7433.198728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7438.821116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7438.824720] Lustre: Skipped 2 previous similar messages [ 7439.154178] LustreError: 476220:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7439.157925] LustreError: 476220:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7439.224865] Lustre: server umount lustre-OST0000 complete [ 7443.424594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7443.429051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7443.436019] Lustre: Skipped 1 previous similar message [ 7443.439234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7447.251773] LustreError: 476422:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7447.255348] LustreError: 476422:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7447.298059] Lustre: server umount lustre-MDT0000 complete [ 7449.057796] LustreError: 469731:0:(ldlm_lib.c:1181:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7449.066880] LustreError: 469731:0:(ldlm_lib.c:1181:target_handle_connect()) Skipped 3 previous similar messages [ 7449.139527] LustreError: 469710:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768147432 with bad export cookie 11526885611628754806 [ 7449.146776] LustreError: 469710:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7449.147470] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7449.331496] Lustre: server umount lustre-MDT0001 complete [ 7454.485142] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7456.105862] Key type lgssc unregistered [ 7456.283687] LNet: 477155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7456.288509] LNetError: 477155:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7456.299731] LNet: Removed LNI 192.168.203.104@tcp [ 7456.792126] Key type .llcrypt unregistered [ 7456.794183] Key type ._llcrypt unregistered [ 7466.097297] Key type ._llcrypt registered [ 7466.098677] Key type .llcrypt registered [ 7466.156648] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7466.668942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7466.696095] alg: No test for adler32 (adler32-zlib) [ 7467.594264] Lustre: Lustre: Build Version: 2.17.50_31_g629c059 [ 7467.694378] LNet: Added LNI 192.168.203.104@tcp [8/256/0/180] [ 7469.279176] Key type lgssc registered [ 7469.927667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7488.671953] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 7493.419600] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7511.092825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7511.111545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7512.262137] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7512.278391] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7512.337511] Lustre: lustre-MDT0000: new disk, initializing [ 7512.378698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7512.391263] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7514.244474] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7519.949067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7520.022352] Lustre: 481537:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7520.068398] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7520.072133] Lustre: Skipped 1 previous similar message [ 7520.123663] Lustre: lustre-MDT0001: new disk, initializing [ 7520.173231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7520.214446] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7520.220558] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7522.119366] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7526.261228] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7528.093004] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7531.433514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7531.597347] Lustre: lustre-OST0000: new disk, initializing [ 7531.599940] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7531.645185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7534.325607] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_default_debug -1 all [ 7538.480502] Lustre: DEBUG MARKER: oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7540.561893] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7540.753411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7540.759832] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7540.825847] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7541.729726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 7543.622486] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7543.749885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7545.830596] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7545.834478] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7545.835323] LustreError: Skipped 1 previous similar message [ 7545.843789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7545.849060] Lustre: Skipped 1 previous similar message [ 7550.773200] LustreError: 484477:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7550.840384] Lustre: server umount lustre-OST0000 complete [ 7556.068081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7556.069036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7556.069458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7556.075579] Lustre: Skipped 2 previous similar messages [ 7558.869519] LustreError: 484678:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7558.873155] LustreError: 484678:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7558.917291] Lustre: server umount lustre-MDT0000 complete [ 7560.780368] LustreError: 481528:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1768147543 with bad export cookie 13506806634750776807 [ 7560.787648] LustreError: 481528:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7560.788073] LustreError: MGC192.168.203.104@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7560.872578] LustreError: 484880:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7560.874957] LustreError: 484880:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7560.950136] Lustre: server umount lustre-MDT0001 complete [ 7572.663344] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing set_hostid [ 7577.539153] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing load_modules_local [ 7593.394654] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7469 sec ========== 11:06:15 (1768147575) [ 7594.247514] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:06:16 (1768147576) === [ 7595.817272] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:06:17 (1768147577) === [ 7614.552085] Lustre: DEBUG MARKER: oleg304-server.virtnet: executing unload_modules_local [ 7616.027152] Key type lgssc unregistered [ 7616.193565] LNet: 489720:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7616.198365] LNetError: 489720:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7616.208549] LNet: Removed LNI 192.168.203.104@tcp [ 7616.641627] Key type .llcrypt unregistered [ 7616.643533] Key type ._llcrypt unregistered