[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 500963320 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002362] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007287] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010134] LSM: Security Framework initializing [ 0.011052] Yama: becoming mindful. [ 0.012039] SELinux: Initializing. [ 0.013052] *** VALIDATE selinux *** [ 0.021303] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026442] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028089] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029125] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031103] *** VALIDATE tmpfs *** [ 0.032445] *** VALIDATE proc *** [ 0.033229] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035258] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036159] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038027] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.042408] debug: unmapping init [mem 0xffffffff8b859000-0xffffffff8b860fff] [ 0.044957] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045602] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049012] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051020] ... fixed-purpose events: 3 [ 0.052010] ... event mask: 000000070000000f [ 0.053302] rcu: Hierarchical SRCU implementation. [ 0.055422] smp: Bringing up secondary CPUs ... [ 0.056565] x86: Booting SMP configuration: [ 0.057026] .... node #0, CPUs: #1 #2 #3 [ 0.060226] smp: Brought up 1 node, 4 CPUs [ 0.062013] smpboot: Max logical packages: 1 [ 0.063017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132554] node 0 deferred pages initialised in 68ms [ 0.135131] devtmpfs: initialized [ 0.137323] x86/mm: Memory block size: 128MB [ 0.140761] gcov: version magic: 0x41383552 [ 0.144323] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.147073] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.149282] pinctrl core: initialized pinctrl subsystem [ 0.150121] [ 0.150596] ************************************************************* [ 0.152010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154008] ** ** [ 0.155007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156007] ** ** [ 0.158013] ** This means that this kernel is built to expose internal ** [ 0.160011] ** IOMMU data structures, which may compromise security on ** [ 0.161008] ** your system. ** [ 0.163008] ** ** [ 0.164008] ** If you see this message and you are not debugging the ** [ 0.165011] ** kernel, report this immediately to your vendor! ** [ 0.167009] ** ** [ 0.169011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170007] ************************************************************* [ 0.172564] NET: Registered protocol family 16 [ 0.173295] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.175036] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.176053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.178447] cpuidle: using governor menu [ 0.180568] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.183483] PCI: Using configuration type 1 for base access [ 0.186141] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.195086] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.198055] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.202110] cryptd: max_cpu_qlen set to 1000 [ 0.206363] ACPI: Added _OSI(Module Device) [ 0.207013] ACPI: Added _OSI(Processor Device) [ 0.209013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.210010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.214237] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.220447] ACPI: Interpreter enabled [ 0.222065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.224013] ACPI: Using IOAPIC for interrupt routing [ 0.226113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.230361] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.240262] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.243052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.246025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.250084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.257119] acpiphp: Slot [2] registered [ 0.258117] acpiphp: Slot [5] registered [ 0.259101] acpiphp: Slot [6] registered [ 0.260089] acpiphp: Slot [7] registered [ 0.261115] acpiphp: Slot [8] registered [ 0.262089] acpiphp: Slot [9] registered [ 0.263092] acpiphp: Slot [10] registered [ 0.265092] acpiphp: Slot [3] registered [ 0.266083] acpiphp: Slot [4] registered [ 0.267093] acpiphp: Slot [11] registered [ 0.268094] acpiphp: Slot [12] registered [ 0.269097] acpiphp: Slot [13] registered [ 0.270080] acpiphp: Slot [14] registered [ 0.271076] acpiphp: Slot [15] registered [ 0.272089] acpiphp: Slot [16] registered [ 0.273075] acpiphp: Slot [17] registered [ 0.274075] acpiphp: Slot [18] registered [ 0.276118] acpiphp: Slot [19] registered [ 0.277094] acpiphp: Slot [20] registered [ 0.278079] acpiphp: Slot [21] registered [ 0.279074] acpiphp: Slot [22] registered [ 0.280080] acpiphp: Slot [23] registered [ 0.281076] acpiphp: Slot [24] registered [ 0.282116] acpiphp: Slot [25] registered [ 0.283083] acpiphp: Slot [26] registered [ 0.284116] acpiphp: Slot [27] registered [ 0.286113] acpiphp: Slot [28] registered [ 0.287093] acpiphp: Slot [29] registered [ 0.288099] acpiphp: Slot [30] registered [ 0.290103] acpiphp: Slot [31] registered [ 0.291082] PCI host bridge to bus 0000:00 [ 0.292018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.295026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.296024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.299022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.301023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.303023] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.304174] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.305949] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.309258] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.318331] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.321457] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.322013] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.323014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.325011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.326443] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.329504] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.330032] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.332559] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.335012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.341897] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.346021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.349827] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.372020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.380023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.399053] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.408255] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.426017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.442017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.478018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.489871] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.503019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.515015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.549020] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.558485] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.581016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.601017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.656155] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.690827] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.705020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.723022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.742017] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.752384] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.777014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.786020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.809015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.829892] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.832366] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.835345] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.837458] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.840229] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.845082] iommu: Default domain type: Passthrough [ 0.846312] SCSI subsystem initialized [ 0.847117] ACPI: bus type USB registered [ 0.848061] usbcore: registered new interface driver usbfs [ 0.850068] usbcore: registered new interface driver hub [ 0.852110] usbcore: registered new device driver usb [ 0.853160] pps_core: LinuxPPS API ver. 1 registered [ 0.855012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.859067] PTP clock support registered [ 0.862045] EDAC MC: Ver: 3.0.0 [ 0.864125] PCI: Using ACPI for IRQ routing [ 0.865864] NetLabel: Initializing [ 0.868012] NetLabel: domain hash size = 128 [ 0.869009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.871079] NetLabel: unlabeled traffic allowed by default [ 0.874311] vgaarb: loaded [ 0.875259] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.877014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.887129] clocksource: Switched to clocksource kvm-clock [ 0.985417] VFS: Disk quotas dquot_6.6.0 [ 0.986473] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.988113] *** VALIDATE ramfs *** [ 0.988923] *** VALIDATE hugetlbfs *** [ 0.990029] pnp: PnP ACPI init [ 0.991727] pnp: PnP ACPI: found 6 devices [ 1.013917] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.017104] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.018372] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.019761] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.021366] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.022837] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.024515] NET: Registered protocol family 2 [ 1.026200] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.030253] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.033796] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.039070] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.042361] TCP: Hash tables configured (established 65536 bind 65536) [ 1.045038] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.048127] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.050382] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.052815] NET: Registered protocol family 1 [ 1.055381] RPC: Registered named UNIX socket transport module. [ 1.057711] RPC: Registered udp transport module. [ 1.059508] RPC: Registered tcp transport module. [ 1.062158] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.064440] NET: Registered protocol family 44 [ 1.066114] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.068058] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.070252] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.072517] PCI: CLS 0 bytes, default 64 [ 1.074157] Unpacking initramfs... [ 2.492740] debug: unmapping init [mem 0xffff8ef4bcc54000-0xffff8ef4bffbffff] [ 2.496602] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.498744] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.501552] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.021159] Initialise system trusted keyrings [ 3.022878] Key type blacklist registered [ 3.024740] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.033404] zbud: loaded [ 3.036161] *** VALIDATE nfs *** [ 3.037290] *** VALIDATE nfs4 *** [ 3.038601] pstore: using deflate compression [ 3.041778] Platform Keyring initialized [ 3.132781] NET: Registered protocol family 38 [ 3.134321] Key type asymmetric registered [ 3.135235] Asymmetric key parser 'x509' registered [ 3.136130] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.137909] io scheduler mq-deadline registered [ 3.138726] io scheduler kyber registered [ 3.139592] io scheduler bfq registered [ 3.140753] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.143061] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.145085] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.146896] ACPI: Power Button [PWRF] [ 3.150643] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.155969] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.166502] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.171693] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.182271] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.208653] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.239872] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.243911] Non-volatile memory driver v1.3 [ 3.245045] Linux agpgart interface v0.103 [ 3.271255] virtio_blk virtio1: [vda] 134360 512-byte logical blocks (68.8 MB/65.6 MiB) [ 3.273566] vda: detected capacity change from 0 to 68792320 [ 3.290316] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.292590] vdb: detected capacity change from 0 to 1073741824 [ 3.305895] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.308110] vdc: detected capacity change from 0 to 2621440000 [ 3.318473] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.320179] vdd: detected capacity change from 0 to 2621440000 [ 3.330492] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.331983] vde: detected capacity change from 0 to 4294967296 [ 3.348475] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.350995] vdf: detected capacity change from 0 to 4294967296 [ 3.361740] libphy: Fixed MDIO Bus: probed [ 3.366667] usbcore: registered new interface driver usbserial_generic [ 3.368534] usbserial: USB Serial support registered for generic [ 3.370105] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.373754] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.375569] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.378170] mousedev: PS/2 mouse device common for all mice [ 3.380921] rtc_cmos 00:05: RTC can wake from S4 [ 3.383210] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.386279] rtc_cmos 00:05: registered as rtc0 [ 3.387587] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.390141] intel_pstate: CPU model not supported [ 3.393463] hid: raw HID events driver (C) Jiri Kosina [ 3.395539] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.396458] usbcore: registered new interface driver usbhid [ 3.396463] usbhid: USB HID core driver [ 3.396573] drop_monitor: Initializing network drop monitor service [ 3.396694] Initializing XFRM netlink socket [ 3.398191] NET: Registered protocol family 10 [ 3.412225] Segment Routing with IPv6 [ 3.413748] NET: Registered protocol family 17 [ 3.415588] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.415894] mpls_gso: MPLS GSO support [ 3.424401] RAS: Correctable Errors collector initialized. [ 3.426155] AVX version of gcm_enc/dec engaged. [ 3.427602] AES CTR mode by8 optimization enabled [ 3.497541] sched_clock: Marking stable (3497524716, 0)->(4369628854, -872104138) [ 3.501564] registered taskstats version 1 [ 3.504311] Loading compiled-in X.509 certificates [ 3.506317] zswap: loaded using pool lzo/zbud [ 3.524693] Key type big_key registered [ 3.534224] Key type encrypted registered [ 3.535174] ima: No TPM chip found, activating TPM-bypass! [ 3.536406] ima: Allocated hash algorithm: sha1 [ 3.537353] ima: No architecture policies found [ 3.538304] evm: Initialising EVM extended attributes: [ 3.539246] evm: security.selinux [ 3.539837] evm: security.ima [ 3.540585] evm: security.capability [ 3.541328] evm: HMAC attrs: 0x1 [ 3.542960] rtc_cmos 00:05: setting system clock to 2026-01-30 06:30:26 UTC (1769754626) [ 3.547217] debug: unmapping init [mem 0xffffffff8c803000-0xffffffff8c9fffff] [ 3.548902] debug: unmapping init [mem 0xffffffff8b582000-0xffffffff8b858fff] [ 3.558129] Write protecting the kernel read-only data: 28672k [ 3.560560] debug: unmapping init [mem 0xffffffff89c03000-0xffffffff89dfffff] [ 3.562264] debug: unmapping init [mem 0xffffffff8a514000-0xffffffff8a5fffff] [ 3.585814] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.590912] systemd[1]: Detected virtualization kvm. [ 3.591954] systemd[1]: Detected architecture x86-64. [ 3.593190] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.622301] systemd[1]: No hostname configured. [ 3.623406] systemd[1]: Set hostname to . [ 3.624807] random: systemd: uninitialized urandom read (16 bytes read) [ 3.626186] systemd[1]: Initializing machine ID from random generator. [ 3.738089] random: systemd: uninitialized urandom read (16 bytes read) [ 3.740886] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.744168] random: systemd: uninitialized urandom read (16 bytes read) [ 3.746036] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.752988] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.222613] device-mapper: uevent: version 1.0.3 [ 4.224564] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.815173] virtio_net virtio0 ens2: renamed from eth0 [ 4.830063] random: fast init done [ 4.873965] scsi host0: ata_piix [ 4.923512] scsi host1: ata_piix [ 4.924695] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.926426] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.393196] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.045689] random: crng init done [ 10.046965] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.253846] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.263398] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.517636] SELinux: Disabled at runtime. [ 11.583488] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.591878] systemd[1]: Detected virtualization kvm. [ 11.593674] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.120403] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.123095] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.126825] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.131312] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.134538] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.140814] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.148583] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 12.285923] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.594722] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.895747] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.931816] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.984457] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.003626] EDAC sbridge: Ver: 1.1.2 [ 14.687507] Key type dns_resolver registered [ 14.986744] NFS: Registering the id_resolver key type [ 14.988627] Key type id_resolver registered [ 14.990189] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg651-server login: [ 110.937316] hrtimer: interrupt took 5374074 ns [ 113.006350] libcfs: loading out-of-tree module taints kernel. [ 113.100175] Key type ._llcrypt registered [ 113.102664] Key type .llcrypt registered [ 113.167881] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 132.029624] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 133.759651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 133.772233] alg: No test for adler32 (adler32-zlib) [ 135.275631] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 136.313535] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 138.146130] Key type lgssc registered [ 139.718360] Lustre: Echo OBD driver; http://www.lustre.org/ [ 159.721519] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 211.219785] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 228.108937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 228.158570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 229.616249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 229.690387] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 229.772542] Lustre: lustre-MDT0000: new disk, initializing [ 229.863713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 229.879488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 236.058711] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 251.401313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 251.574315] Lustre: 6525:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 251.629466] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 251.634315] Lustre: Skipped 1 previous similar message [ 251.780373] Lustre: lustre-MDT0001: new disk, initializing [ 251.897082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 251.955430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 251.984717] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 257.042529] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 262.757379] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 273.776293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 274.102414] Lustre: lustre-OST0000: new disk, initializing [ 274.107984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 274.210767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 281.467938] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 283.701244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 283.706618] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 283.772392] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 298.106505] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 298.333626] Lustre: lustre-OST0001: new disk, initializing [ 298.342361] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 298.463431] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 304.216335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 304.233369] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 304.353870] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 304.948194] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 317.316452] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 323.391973] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 336.018806] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing check_logdir /tmp/testlogs/ [ 342.133868] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing yml_node [ 346.141055] Lustre: DEBUG MARKER: Client: 2.17.50.86 [ 348.839611] Lustre: DEBUG MARKER: MDS: 2.17.50.86 [ 351.678682] Lustre: DEBUG MARKER: OSS: 2.17.50.86 [ 353.308624] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jan 30 01:36:15 EST 2026 [ 373.205866] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 375.060310] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 395.233573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 395.244669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 395.261758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 396.779499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 396.788262] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 396.792021] Lustre: Skipped 2 previous similar messages [ 396.821079] Lustre: Skipped 2 previous similar messages [ 397.957391] LustreError: 12227:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 398.127122] Lustre: server umount lustre-MDT0000 complete [ 402.651051] LustreError: 6518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755025 with bad export cookie 12810592157084123480 [ 402.654575] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 402.678036] LustreError: 6518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 407.009684] LustreError: 10012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 407.009904] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 407.010457] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 407.013055] Lustre: Skipped 1 previous similar message [ 407.047363] LustreError: 10012:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 407.096776] Lustre: Skipped 1 previous similar message [ 408.931649] LustreError: 12430:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 408.954600] LustreError: 12430:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 409.200345] Lustre: server umount lustre-MDT0001 complete [ 414.163123] LustreError: 12633:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 414.172779] LustreError: 12633:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 414.356158] Lustre: server umount lustre-OST0000 complete [ 418.261075] LustreError: 12833:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 418.269584] LustreError: 12833:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 418.398763] Lustre: server umount lustre-OST0001 complete [ 426.202372] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 437.010146] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 479.610200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 479.889717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 479.935833] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 480.014376] Lustre: lustre-MDT0000: new disk, initializing [ 480.103731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 480.143087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 485.706668] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 496.687689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 496.802856] Lustre: 15006:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 496.864475] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 496.868153] Lustre: Skipped 1 previous similar message [ 496.957293] Lustre: lustre-MDT0001: new disk, initializing [ 497.083288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 497.141394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 497.160048] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 501.310634] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 509.777711] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 514.717592] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 523.087842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 523.597323] Lustre: lustre-OST0000: new disk, initializing [ 523.608596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 523.738829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 525.300451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 525.321609] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 525.552415] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 532.029355] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 540.444668] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 544.765786] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 544.924588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 549.026784] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 549.263871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 556.003115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 556.014054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.044322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 556.058487] Lustre: Skipped 1 previous similar message [ 561.639975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 561.647646] Lustre: Skipped 2 previous similar messages [ 566.240122] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 566.295284] LustreError: 17939:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 566.300415] LustreError: 17939:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 566.671651] Lustre: server umount lustre-OST0000 complete [ 571.875716] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 571.890560] Lustre: Skipped 1 previous similar message [ 571.895249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 571.898955] Lustre: Skipped 1 previous similar message [ 576.739819] LustreError: 18142:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 576.744700] LustreError: 18142:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 576.947455] Lustre: server umount lustre-MDT0000 complete [ 576.993700] LustreError: 15017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 577.019474] LustreError: 15017:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 580.361424] LustreError: 15784:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755203 with bad export cookie 12810592157084125013 [ 580.364055] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 580.378205] LustreError: 15784:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 580.883948] Lustre: server umount lustre-MDT0001 complete [ 585.373172] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:40:07 (1769755207) [ 588.251713] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 590.623326] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:40:12 (1769755212) [ 598.660571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 599.454615] LustreError: 19013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 599.552253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 604.631398] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 604.643563] LustreError: 19013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 609.786066] LustreError: 19013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 615.439578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 621.370275] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 627.692779] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 632.440604] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 641.160902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 650.352759] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 657.779634] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 670.697378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 670.710059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 670.726361] Lustre: Skipped 2 previous similar messages [ 670.734591] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 670.742459] Lustre: Skipped 1 previous similar message [ 670.754536] LustreError: Skipped 1 previous similar message [ 675.394581] LustreError: 21254:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 675.402025] LustreError: 21254:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 675.504709] Lustre: server umount lustre-OST0000 complete [ 686.648863] Lustre: server umount lustre-MDT0000 complete [ 690.163353] LustreError: 19007:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 690.190656] LustreError: 19007:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 691.482244] LustreError: 18992:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755314 with bad export cookie 12810592157084125895 [ 691.486677] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 691.499768] LustreError: 18992:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 692.073150] Lustre: server umount lustre-MDT0001 complete [ 708.484499] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 712.903381] Key type lgssc unregistered [ 713.211402] LNet: 22197:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 713.219936] LNetError: 22197:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 713.243768] LNet: Removed LNI 192.168.206.151@tcp [ 714.569239] Key type .llcrypt unregistered [ 714.574263] Key type ._llcrypt unregistered [ 736.745636] Key type ._llcrypt registered [ 736.751985] Key type .llcrypt registered [ 736.894688] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 738.140433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 738.179035] alg: No test for adler32 (adler32-zlib) [ 739.505345] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 739.845632] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 741.498351] Key type lgssc registered [ 742.760532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 751.931417] loop: module loaded [ 752.034168] loop0: detected capacity change from 0 to 204800000 [ 752.302731] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 752.316252] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 752.575574] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 759.209704] loop0: detected capacity change from 0 to 204800000 [ 759.340383] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 759.351993] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 759.365375] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 764.756662] loop0: detected capacity change from 0 to 204800000 [ 772.046756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 772.069525] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 773.496056] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 773.576735] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 773.652327] Lustre: lustre-MDT0000: new disk, initializing [ 773.725901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 773.739186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 778.414625] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 789.333538] loop1: detected capacity change from 0 to 204800000 [ 796.229846] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 796.824430] Lustre: lustre-OST07c6: new disk, initializing [ 796.830773] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 796.835526] Lustre: Skipped 1 previous similar message [ 796.942684] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 804.439764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 804.444935] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 804.481877] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 805.519562] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 811.107431] Lustre: 24348:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 811.129790] Lustre: 24348:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 811.138325] Lustre: 24348:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 811.150184] Lustre: 24348:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 811.163025] Lustre: 24348:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 811.178077] Lustre: 24348:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 828.774050] Lustre: 25990:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 828.778559] Lustre: 25990:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 828.784303] Lustre: 25990:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 828.799402] Lustre: 25990:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 828.817300] Lustre: 25990:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 828.824440] Lustre: 25990:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 834.655362] Lustre: 25990:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 834.675505] Lustre: 25990:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 834.682984] Lustre: 25990:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 834.691331] Lustre: 25990:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 1 previous similar message [ 834.699921] Lustre: 25990:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 834.707433] Lustre: 25990:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 1 previous similar message [ 834.713254] Lustre: 25990:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 834.719159] Lustre: 25990:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 1 previous similar message [ 834.723463] Lustre: 25990:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 834.729419] Lustre: 25990:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 1 previous similar message [ 834.735666] Lustre: 25990:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 834.745283] Lustre: 25990:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 1 previous similar message [ 840.220191] Lustre: 24348:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 840.229250] Lustre: 24348:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 840.234949] Lustre: 24348:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 840.245273] Lustre: 24348:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 1 previous similar message [ 840.260895] Lustre: 24348:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 840.271593] Lustre: 24348:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 1 previous similar message [ 840.278520] Lustre: 24348:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 840.284893] Lustre: 24348:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 1 previous similar message [ 840.295855] Lustre: 24348:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 840.302591] Lustre: 24348:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 1 previous similar message [ 840.310989] Lustre: 24348:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 840.327748] Lustre: 24348:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 1 previous similar message [ 845.423464] Lustre: 24349:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 845.431051] Lustre: 24349:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 1 previous similar message [ 845.448925] Lustre: 24349:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 845.462220] Lustre: 24349:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 1 previous similar message [ 845.468738] Lustre: 24349:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 845.475296] Lustre: 24349:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 1 previous similar message [ 845.482395] Lustre: 24349:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 845.494193] Lustre: 24349:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 1 previous similar message [ 845.503368] Lustre: 24349:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 845.522559] Lustre: 24349:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 1 previous similar message [ 845.534600] Lustre: 24349:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 845.543377] Lustre: 24349:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 1 previous similar message [ 857.113528] Lustre: 24349:0:(osd_internal.h:1455:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 857.121896] Lustre: 24349:0:(osd_internal.h:1455:osd_trans_exec_op()) Skipped 5 previous similar messages [ 857.131188] Lustre: 24349:0:(osd_handler.c:2093:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 857.145602] Lustre: 24349:0:(osd_handler.c:2093:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 857.154354] Lustre: 24349:0:(osd_handler.c:2100:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 857.165038] Lustre: 24349:0:(osd_handler.c:2100:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 857.173283] Lustre: 24349:0:(osd_handler.c:2110:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 857.179191] Lustre: 24349:0:(osd_handler.c:2110:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 857.191813] Lustre: 24349:0:(osd_handler.c:2117:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 857.195600] Lustre: 24349:0:(osd_handler.c:2117:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 857.201316] Lustre: 24349:0:(osd_handler.c:2124:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 857.206502] Lustre: 24349:0:(osd_handler.c:2124:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 906.735622] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 906.751672] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 906.787140] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 909.800515] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 914.914508] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 919.008202] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 919.059130] LustreError: 26530:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 919.279171] Lustre: server umount lustre-OST07c6 complete [ 930.218197] LustreError: 27033:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 930.226815] LustreError: 27033:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 930.552767] Lustre: server umount lustre-MDT0000 complete [ 951.324941] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 954.315833] Key type lgssc unregistered [ 954.657312] LNet: 27868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.664971] LNetError: 27868:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 954.688901] LNet: Removed LNI 192.168.206.151@tcp [ 955.689550] Key type .llcrypt unregistered [ 955.692520] Key type ._llcrypt unregistered [ 975.562623] Key type ._llcrypt registered [ 975.565344] Key type .llcrypt registered [ 975.661113] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 976.911915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 977.024116] alg: No test for adler32 (adler32-zlib) [ 978.208628] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 978.496521] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 980.192248] Key type lgssc registered [ 981.716382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 990.259826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 990.282620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 991.848223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 997.999733] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1001.462086] LustreError: 29118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1001.490098] LustreError: 29118:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1006.563095] LustreError: 29118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1008.594216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1009.054823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1014.622977] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1021.353415] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1025.929315] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1033.705688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1034.192690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1036.288731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1041.357480] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1048.304121] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1069.028833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1069.032427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1069.038957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1069.049579] Lustre: Skipped 1 previous similar message [ 1073.358345] LustreError: 31572:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1073.700241] Lustre: server umount lustre-OST0000 complete [ 1079.270148] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1079.276103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1079.296729] Lustre: Skipped 1 previous similar message [ 1079.323040] Lustre: Skipped 2 previous similar messages [ 1084.395657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1084.408833] Lustre: Skipped 1 previous similar message [ 1084.789654] LustreError: 31774:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1084.800374] LustreError: 31774:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1084.929907] Lustre: server umount lustre-MDT0000 complete [ 1089.045199] LustreError: 29098:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755712 with bad export cookie 4440873002628036863 [ 1089.047332] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1089.051103] LustreError: 29098:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1089.206357] LustreError: 31975:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1089.218149] LustreError: 31975:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1089.578099] Lustre: server umount lustre-MDT0001 complete [ 1101.115676] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 1104.195535] Key type lgssc unregistered [ 1104.555218] LNet: 32507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1104.577148] LNetError: 32507:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1105.655441] LNet: Removed LNI 192.168.206.151@tcp [ 1106.555242] Key type .llcrypt unregistered [ 1106.557282] Key type ._llcrypt unregistered [ 1125.846661] Key type ._llcrypt registered [ 1125.848910] Key type .llcrypt registered [ 1125.942130] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 1126.957396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1127.033991] alg: No test for adler32 (adler32-zlib) [ 1128.082910] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 1128.297663] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 1129.944300] Key type lgssc registered [ 1131.006777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1138.746540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1138.769970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1140.409566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1143.660304] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1150.948489] LustreError: 33757:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1150.967904] LustreError: 33757:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1152.413840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1152.963443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1158.092555] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1163.928541] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1168.609166] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1175.114854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1175.616653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1181.315336] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1185.873651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1186.980406] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1198.064014] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1198.070207] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1198.085821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1201.121601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1201.138821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1203.596119] LustreError: 36002:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1203.816366] Lustre: server umount lustre-OST0000 complete [ 1209.318545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1209.333170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1209.345234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1209.352128] Lustre: Skipped 1 previous similar message [ 1211.375891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1211.378406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1213.641708] LustreError: 36204:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1213.646976] LustreError: 36204:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1213.791937] Lustre: server umount lustre-MDT0000 complete [ 1216.484288] LustreError: 35923:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1216.498845] LustreError: 35923:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1217.584495] LustreError: 33738:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755840 with bad export cookie 7332684666840559546 [ 1217.588105] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1217.593912] LustreError: 33738:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1217.745156] LustreError: 36406:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1217.752929] LustreError: 36406:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1217.929067] Lustre: server umount lustre-MDT0001 complete [ 1229.354980] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 1233.020724] Key type lgssc unregistered [ 1233.367581] LNet: 36938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1233.375547] LNetError: 36938:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1233.409412] LNet: Removed LNI 192.168.206.151@tcp [ 1234.512765] Key type .llcrypt unregistered [ 1234.515110] Key type ._llcrypt unregistered [ 1265.017485] Key type ._llcrypt registered [ 1265.023649] Key type .llcrypt registered [ 1265.187907] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 1266.203900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1266.278640] alg: No test for adler32 (adler32-zlib) [ 1267.344187] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 1267.593984] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 1269.264156] Key type lgssc registered [ 1270.568737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1278.826784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1278.849699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1280.274401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1284.375645] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1290.727710] LustreError: 38339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1290.753615] LustreError: 38339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1292.587782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1292.875622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1297.476671] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1302.302493] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1305.787028] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1313.001516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1313.265950] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1318.634246] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1323.736749] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1325.591997] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1335.690024] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 1335.779883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1335.796843] Lustre: Skipped 2 previous similar messages [ 1336.951332] LustreError: 40582:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1337.052017] Lustre: server umount lustre-MDT0000 complete [ 1340.377035] LustreError: 38319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769755963 with bad export cookie 9602218659105445100 [ 1340.382154] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1340.385992] LustreError: 38319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1340.775972] LustreError: 38333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1340.823551] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.51@tcp (stopping) [ 1340.828602] Lustre: Skipped 3 previous similar messages [ 1340.898589] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1345.894797] LustreError: 40210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1345.894815] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.51@tcp (stopping) [ 1345.894823] Lustre: Skipped 1 previous similar message [ 1345.922775] LustreError: 40210:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1346.605323] LustreError: 40785:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1346.611330] LustreError: 40785:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1346.798470] Lustre: server umount lustre-MDT0001 complete [ 1367.628693] LustreError: 41012:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1367.639572] LustreError: 41012:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1367.857657] Lustre: server umount lustre-OST0000 complete [ 1386.264751] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 1389.801576] Key type lgssc unregistered [ 1390.096578] LNet: 41850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1390.107142] LNetError: 41850:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1390.127063] LNet: Removed LNI 192.168.206.151@tcp [ 1391.216758] Key type .llcrypt unregistered [ 1391.221185] Key type ._llcrypt unregistered [ 1415.368318] Key type ._llcrypt registered [ 1415.373255] Key type .llcrypt registered [ 1415.523531] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 1417.235148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1417.302843] alg: No test for adler32 (adler32-zlib) [ 1418.409640] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 1418.706928] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 1420.400328] Key type lgssc registered [ 1422.250428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1432.352615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1432.393489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1433.894928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1440.225218] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1443.304649] LustreError: 43247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1443.326293] LustreError: 43247:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1448.417637] LustreError: 43247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1450.209185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1450.733152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1455.282369] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1461.616210] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1465.972859] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1474.980094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1475.365884] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1478.459382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1482.387095] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1488.682713] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1495.451614] Lustre: lustre-OST0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 1497.068815] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1497.078728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1497.106991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1499.110032] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1499.110164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1501.434436] LustreError: 45491:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1501.569528] Lustre: server umount lustre-OST0000 complete [ 1522.664145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1522.675843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1522.704514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1522.714662] Lustre: Skipped 2 previous similar messages [ 1524.714666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1526.009685] LustreError: 45869:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1526.019373] LustreError: 45869:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1526.109295] Lustre: server umount lustre-MDT0000 complete [ 1529.803802] LustreError: 43227:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756152 with bad export cookie 11878972344626416088 [ 1529.819493] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1544.160798] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1544.300859] LustreError: 46070:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1544.306249] LustreError: 46070:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1544.443685] Lustre: server umount lustre-MDT0001 complete [ 1557.145489] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 1559.745132] Key type lgssc unregistered [ 1559.984576] LNet: 46606:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1559.989283] LNetError: 46606:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1561.062736] LNet: Removed LNI 192.168.206.151@tcp [ 1561.661476] Key type .llcrypt unregistered [ 1561.665602] Key type ._llcrypt unregistered [ 1588.916536] Key type ._llcrypt registered [ 1588.926087] Key type .llcrypt registered [ 1589.032427] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 1590.330689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1590.352096] alg: No test for adler32 (adler32-zlib) [ 1591.542552] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 1591.837434] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 1593.576234] Key type lgssc registered [ 1594.657472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1603.228530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1603.252316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1604.698417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1609.670401] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1614.817850] LustreError: 48005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1614.833340] LustreError: 48005:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1619.941362] LustreError: 48005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1620.248520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1620.877961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1625.531111] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1631.971127] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1637.301476] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1645.270993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1645.560577] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 1645.667851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1649.776821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1651.369385] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1657.471337] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1664.523931] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1668.638582] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1680.892435] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-01-30 1h58m22s [ 1682.586528] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1685.880738] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 1685.987658] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1686.007869] Lustre: Skipped 2 previous similar messages [ 1686.959996] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 1686.968992] Lustre: Skipped 3 previous similar messages [ 1690.689269] LustreError: 50568:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1690.846233] Lustre: server umount lustre-MDT0000 complete [ 1692.018726] LustreError: 47999:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1694.606690] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1698.167834] LustreError: 48000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1698.198397] LustreError: 48000:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1701.723036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1701.834885] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1702.065677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1702.131802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1703.104140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1707.079823] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1711.026521] Lustre: DEBUG MARKER: Wait for df (36073) ... [ 1712.883615] Lustre: DEBUG MARKER: done [ 1717.732044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1717.754670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1717.769576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1717.780916] Lustre: Skipped 3 previous similar messages [ 1718.765136] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1723.879829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1723.901496] Lustre: Skipped 3 previous similar messages [ 1731.040140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1731.091309] LustreError: 51628:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1731.100312] LustreError: 51628:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1731.300248] Lustre: server umount lustre-OST0000 complete [ 1739.236866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1739.244870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1739.256554] Lustre: Skipped 1 previous similar message [ 1739.257749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1739.277806] Lustre: Skipped 3 previous similar messages [ 1740.841165] LustreError: 51830:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1740.848486] LustreError: 51830:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1741.038754] Lustre: server umount lustre-MDT0000 complete [ 1744.837352] LustreError: 50298:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756367 with bad export cookie 12773705000041304170 [ 1744.855643] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1744.857207] LustreError: 50298:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1745.019887] LustreError: 52031:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1745.029687] LustreError: 52031:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1745.401822] Lustre: server umount lustre-MDT0001 complete [ 1756.995325] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 1760.344663] Key type lgssc unregistered [ 1760.673051] LNet: 52564:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1760.682266] LNetError: 52564:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1760.700331] LNet: Removed LNI 192.168.206.151@tcp [ 1761.529410] Key type .llcrypt unregistered [ 1761.531626] Key type ._llcrypt unregistered [ 1811.497669] Key type ._llcrypt registered [ 1811.501682] Key type .llcrypt registered [ 1811.770794] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 1813.249906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1813.298052] alg: No test for adler32 (adler32-zlib) [ 1814.613730] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 1814.833051] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 1816.512192] Key type lgssc registered [ 1818.431705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1825.912600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1825.930430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1827.173759] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1827.201977] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1827.354666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1831.558164] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1841.498654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1841.657785] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1841.710089] Lustre: 54751:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1842.043058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1847.441909] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1853.332935] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1857.042676] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1864.578382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1864.793758] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1864.987099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1871.294897] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 1871.513679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1877.012327] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1884.559943] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-01-30 2h01m46s [ 1885.861458] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1892.121554] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1894.717566] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1895.057846] LustreError: 54758:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1915.271985] Lustre: lustre-MDT0000: Client dad919ab-d06b-4801-adb1-44030d7ccda4 (at 192.168.206.51@tcp) reconnecting [ 1935.064115] LustreError: 54758:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1937.231822] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1939.836979] Lustre: DEBUG MARKER: done [ 1948.129416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1948.132834] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1948.148247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1948.643869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1948.658164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1952.403056] LustreError: 57803:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1952.715568] Lustre: server umount lustre-OST0000 complete [ 1958.882228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1958.888388] Lustre: Skipped 1 previous similar message [ 1958.889105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1958.896852] Lustre: Skipped 1 previous similar message [ 1962.807508] LustreError: 58004:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1962.813640] LustreError: 58004:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1963.222287] Lustre: server umount lustre-MDT0000 complete [ 1967.053450] LustreError: 54745:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756590 with bad export cookie 11967176428024783286 [ 1967.060493] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1967.067647] LustreError: 54745:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1967.241208] LustreError: 58205:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1967.247929] LustreError: 58205:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1967.477836] Lustre: server umount lustre-MDT0001 complete [ 1979.131024] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 1981.862651] Key type lgssc unregistered [ 1982.118719] LNet: 58738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1982.128488] LNetError: 58738:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1982.144869] LNet: Removed LNI 192.168.206.151@tcp [ 1983.090235] Key type .llcrypt unregistered [ 1983.095798] Key type ._llcrypt unregistered [ 2029.087989] Key type ._llcrypt registered [ 2029.090123] Key type .llcrypt registered [ 2029.208264] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2030.020842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2030.054306] alg: No test for adler32 (adler32-zlib) [ 2031.192593] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 2031.474025] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 2033.120637] Key type lgssc registered [ 2034.234867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2037.306062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2037.324958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2038.732788] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2038.762583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2038.935340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2039.897454] Lustre: Failing over lustre-MDT0000 [ 2039.976721] LustreError: 60962:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2040.057441] Lustre: server umount lustre-MDT0000 complete [ 2046.728612] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 02:04:28 (1769756668) [ 2054.545288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2055.151300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2059.313407] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2068.933573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2069.084797] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2069.116493] Lustre: 61516:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2069.379790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2073.202796] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2078.541350] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2083.214691] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2089.894341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2090.115046] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2090.335651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2096.600353] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2098.346857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2103.441723] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2111.108495] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2119.138221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2119.140859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2119.155937] Lustre: Skipped 2 previous similar messages [ 2119.171873] Lustre: Skipped 2 previous similar messages [ 2124.258572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2124.275315] Lustre: Skipped 2 previous similar messages [ 2124.576519] LustreError: 63885:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2124.588263] LustreError: 63885:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2124.646040] Lustre: server umount lustre-MDT0000 complete [ 2128.072334] LustreError: 63144:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756751 with bad export cookie 8189834974760795845 [ 2128.076572] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2128.085840] LustreError: 63144:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2128.198062] LustreError: 64086:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2128.201835] LustreError: 64086:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2128.428249] Lustre: server umount lustre-MDT0001 complete [ 2132.088900] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2137.474234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2145.508814] Lustre: 60440:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769756752/real 1769756752] req@00000000ea0f1fa3 x1855724353265152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769756768 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2145.553917] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2147.557618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2149.856469] Lustre: 60442:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769756756/real 1769756756] req@0000000037de0b35 x1855724353265280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769756772 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2150.880221] Lustre: 60441:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769756757/real 1769756757] req@000000003b4dad5e x1855724353265408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769756773 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2153.760891] LustreError: 60438:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000faedecdf x1855724353266176/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2154.327145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2154.499109] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2155.296145] Lustre: 60439:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769756761/real 1769756761] req@00000000ec8f29e6 x1855724353265664/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769756777 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2159.144858] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2159.456271] Lustre: 60439:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769756766/real 1769756766] req@000000000a345eb5 x1855724353265792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769756782 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2160.623134] LustreError: 64947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2160.642548] LustreError: 64947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2165.733045] LustreError: 64946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2165.748203] LustreError: 64946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2168.611236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2169.042072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2174.144717] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2174.194442] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2181.149765] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2185.604662] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2194.913930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2194.923208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2194.924965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2194.944147] Lustre: Skipped 2 previous similar messages [ 2194.969449] Lustre: Skipped 2 previous similar messages [ 2200.043318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2200.053816] Lustre: Skipped 1 previous similar message [ 2200.243796] LustreError: 66347:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2200.254542] LustreError: 66347:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2200.511873] Lustre: server umount lustre-MDT0000 complete [ 2205.163901] LustreError: 64946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2205.199867] LustreError: 64946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2205.289971] LustreError: 61510:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756828 with bad export cookie 8189834974760798120 [ 2205.295724] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2205.296724] LustreError: 61510:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2205.513903] LustreError: 66549:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2205.534714] LustreError: 66549:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2205.756359] Lustre: server umount lustre-MDT0001 complete [ 2213.453192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2225.100978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2226.656509] Lustre: 60442:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769756833/real 1769756833] req@0000000025cafc95 x1855724353293056/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769756849 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2226.681565] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2230.822648] LustreError: 60438:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009ad66adf x1855724353293824/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2231.134045] LustreError: 63139:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2231.219969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2231.269178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2235.298755] LustreError: 63139:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2235.336636] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2235.354502] Lustre: Skipped 1 previous similar message [ 2236.167878] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2246.367233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2246.629043] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2251.528786] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2252.276475] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2259.170655] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2263.914978] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2272.225649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2272.231780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.251589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2272.263577] Lustre: Skipped 1 previous similar message [ 2277.881795] LustreError: 68808:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2277.884637] LustreError: 68808:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2278.109192] Lustre: server umount lustre-MDT0000 complete [ 2282.790177] LustreError: 61509:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769756905 with bad export cookie 8189834974760799499 [ 2282.801983] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2282.802776] LustreError: 61509:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2282.977431] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2282.979908] LustreError: 67408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2282.991976] Lustre: Skipped 2 previous similar messages [ 2283.005052] LustreError: 67408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2283.433322] Lustre: server umount lustre-MDT0001 complete [ 2290.097910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2294.204343] Lustre: DEBUG MARKER: files compared the same [ 2296.573994] LustreError: 69603:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2296.586542] LustreError: 69603:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2296.795957] Lustre: server umount lustre-OST0000 complete [ 2315.461853] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 2318.174966] Key type lgssc unregistered [ 2318.437703] LNet: 70442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2318.443420] LNetError: 70442:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2318.472258] LNet: Removed LNI 192.168.206.151@tcp [ 2319.467258] Key type .llcrypt unregistered [ 2319.468770] Key type ._llcrypt unregistered [ 2343.280592] Key type ._llcrypt registered [ 2343.284594] Key type .llcrypt registered [ 2343.384225] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2344.756692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2344.807626] alg: No test for adler32 (adler32-zlib) [ 2345.960492] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 2346.144222] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 2347.824276] Key type lgssc registered [ 2349.073633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2356.730892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2356.745622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2358.307390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2362.749127] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug +malloc all [ 2368.480863] LustreError: 71838:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2368.504591] LustreError: 71838:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2371.135033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2371.617378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2375.894844] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug +malloc all [ 2382.629326] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2387.054131] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2395.099679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2395.537982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2402.844658] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug +malloc all [ 2403.827401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2409.549738] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2419.169924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2419.179156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2419.192879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2424.128618] LustreError: 74079:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2424.268995] Lustre: server umount lustre-OST0000 complete [ 2429.409311] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2429.418482] Lustre: Skipped 1 previous similar message [ 2429.431976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2429.440332] Lustre: Skipped 2 previous similar messages [ 2434.531475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2434.539617] Lustre: Skipped 1 previous similar message [ 2434.926701] LustreError: 74281:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2434.940653] LustreError: 74281:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2435.346692] Lustre: server umount lustre-MDT0000 complete [ 2439.046761] LustreError: 71818:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769757061 with bad export cookie 6989367491705912226 [ 2439.056848] LustreError: 71818:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2439.057244] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2439.248112] LustreError: 74482:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2439.253171] LustreError: 74482:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2439.420888] Lustre: server umount lustre-MDT0001 complete [ 2451.231127] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 2454.216396] Key type lgssc unregistered [ 2454.530578] LNet: 75015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2454.538982] LNetError: 75015:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2454.557489] LNet: Removed LNI 192.168.206.151@tcp [ 2455.215892] Key type .llcrypt unregistered [ 2455.222107] Key type ._llcrypt unregistered [ 2479.445758] Key type ._llcrypt registered [ 2479.449379] Key type .llcrypt registered [ 2479.539921] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2480.663588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2480.818028] alg: No test for adler32 (adler32-zlib) [ 2482.203573] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 2482.418289] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 2484.136576] Key type lgssc registered [ 2485.411269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2493.862439] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2493.923061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2509.536380] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2514.656365] LustreError: 76391:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.151@tcp: failed processing log, type 4: rc = -110 [ 2541.408182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2547.937262] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2553.806360] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2561.994840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2562.123439] Lustre: *** cfs_fail_loc=706, val=0*** [ 2562.600384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2562.696408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2566.230972] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2567.658719] LustreError: 76417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2567.675327] LustreError: 76417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2572.772161] LustreError: 76417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2574.594288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2574.854338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2578.314350] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2583.492899] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2587.286396] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2590.178521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2590.190455] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2590.211441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2592.238511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2592.241424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2595.605763] LustreError: 78714:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2595.750628] Lustre: server umount lustre-OST0000 complete [ 2600.420773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2600.436551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2600.450067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2600.452784] Lustre: Skipped 1 previous similar message [ 2602.467669] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2602.469558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2602.488782] Lustre: Skipped 1 previous similar message [ 2605.689968] LustreError: 78918:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2605.702043] LustreError: 78918:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2605.844877] Lustre: server umount lustre-MDT0000 complete [ 2607.587397] LustreError: 77370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2607.597085] LustreError: 77370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2609.457850] LustreError: 76398:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769757232 with bad export cookie 9732814227833373280 [ 2609.465430] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2609.469447] LustreError: 76398:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2609.603935] LustreError: 79119:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2609.619993] LustreError: 79119:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2609.924811] Lustre: server umount lustre-MDT0001 complete [ 2619.523128] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 2621.884401] Key type lgssc unregistered [ 2622.101578] LNet: 79651:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2622.105839] LNetError: 79651:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2622.124150] LNet: Removed LNI 192.168.206.151@tcp [ 2622.897189] Key type .llcrypt unregistered [ 2622.898874] Key type ._llcrypt unregistered [ 2646.280832] Key type ._llcrypt registered [ 2646.282276] Key type .llcrypt registered [ 2646.364659] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2647.520461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2647.572299] alg: No test for adler32 (adler32-zlib) [ 2648.699865] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 2648.951928] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 2650.688206] Key type lgssc registered [ 2651.872510] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2659.742345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2659.770367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2665.127296] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2671.942941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2672.523293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2676.232511] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2677.732737] LustreError: 81712:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.743408] LustreError: 81712:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2682.852377] LustreError: 81715:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2682.878445] LustreError: 81715:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2684.391272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2684.628382] LustreError: 81711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2684.649968] LustreError: 81711:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2684.676643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2690.022255] LustreError: 81716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2690.027909] LustreError: 81716:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2690.422634] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2695.146863] LustreError: 81716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2695.171777] LustreError: 81716:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2697.884831] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2697.897848] Lustre: MGS: Client c4856630-f31a-49b0-b2e6-8336e48bf731 (at 0@lo) reconnecting [ 2697.914683] LustreError: 81726:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2698.117521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2698.219795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2702.095096] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2718.696034] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2718.699916] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2718.713164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2721.297213] LustreError: 83672:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2721.460207] Lustre: server umount lustre-OST0000 complete [ 2728.929744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2728.942853] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2728.949342] LustreError: Skipped 1 previous similar message [ 2728.951533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2728.968032] Lustre: Skipped 2 previous similar messages [ 2728.982799] Lustre: Skipped 2 previous similar messages [ 2731.469901] LustreError: 83873:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2731.486565] LustreError: 83873:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2731.592651] Lustre: server umount lustre-MDT0000 complete [ 2734.053650] LustreError: 81716:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2734.080570] LustreError: 81716:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2735.812591] LustreError: 84076:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2735.818859] LustreError: 84076:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2736.145581] Lustre: server umount lustre-MDT0001 complete [ 2740.269224] LustreError: 84276:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2740.273716] LustreError: 84276:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2740.318804] Lustre: server umount MGS complete [ 2752.912483] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 02:16:14 (1769757374) [ 2783.838944] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 2793.120885] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 2829.525627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2834.553871] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2846.705970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2846.908330] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2846.955116] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2847.065116] Lustre: lustre-MDT0001: new disk, initializing [ 2847.163925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2847.195496] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2847.200633] Lustre: 88781:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2847.212048] LustreError: 88781:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2852.006672] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2863.511854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2863.756683] Lustre: lustre-OST0000: new disk, initializing [ 2863.761729] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2863.852934] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2870.104339] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2872.929673] Lustre: 89955:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000005bcfeb22 x1855725000236288/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1769757511 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2877.207838] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2884.071835] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2884.091402] Lustre: MGS: Client 7113be7f-b616-470d-a290-9e4da06eb963 (at 0@lo) reconnecting [ 2884.155538] Lustre: 90463:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2884.177206] LustreError: 88767:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2884.188781] LustreError: 88767:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2884.217319] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2884.322420] Lustre: lustre-MDT0000: new disk, initializing [ 2884.543120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2884.596435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2884.758298] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2885.668053] Lustre: 90490:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000080f4e5d9 x1855725000242816/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1769757524 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2889.411863] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 2893.310620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2893.324488] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2894.775304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2894.784087] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2894.980442] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2920.933265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2920.947597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2920.965596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2925.325103] LustreError: 91015:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2925.328658] LustreError: 91015:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2925.574385] Lustre: server umount lustre-OST0000 complete [ 2932.706504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2932.710375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2932.730276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2932.746780] Lustre: Skipped 2 previous similar messages [ 2932.773904] Lustre: Skipped 3 previous similar messages [ 2935.437984] LustreError: 91217:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2935.455682] LustreError: 91217:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2935.508127] Lustre: server umount lustre-MDT0000 complete [ 2937.825525] LustreError: 88751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2937.844443] LustreError: 88751:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2946.085138] Lustre: server umount lustre-MDT0001 complete [ 2951.033631] Lustre: server umount MGS complete [ 2963.453253] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 02:19:45 (1769757585) [ 2981.087415] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 2983.745824] Key type lgssc unregistered [ 2984.139587] LNet: 92959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2984.147438] LNetError: 92959:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2984.167044] LNet: Removed LNI 192.168.206.151@tcp [ 2984.983451] Key type .llcrypt unregistered [ 2984.988047] Key type ._llcrypt unregistered [ 3003.024705] Key type ._llcrypt registered [ 3003.029227] Key type .llcrypt registered [ 3003.201964] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3004.822344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3004.853754] alg: No test for adler32 (adler32-zlib) [ 3005.987165] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 3006.263789] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 3007.944993] Key type lgssc registered [ 3009.457496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3017.884673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3017.922205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3019.428176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3023.587362] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3029.989692] LustreError: 94161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3030.013693] LustreError: 94161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3030.919619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3031.207651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3034.988150] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3039.761150] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3042.961771] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3049.125783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3049.456367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3056.168840] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3062.274851] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3064.844924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3075.044138] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3075.065256] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3075.066754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3075.089946] Lustre: Skipped 1 previous similar message [ 3079.866353] LustreError: 96402:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3079.993215] Lustre: server umount lustre-OST0000 complete [ 3085.282266] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3085.283825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3085.311063] Lustre: Skipped 2 previous similar messages [ 3085.322180] Lustre: Skipped 1 previous similar message [ 3090.314806] LustreError: 96605:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3090.319614] LustreError: 96605:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3090.402408] LustreError: 94158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3090.412834] LustreError: 94158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3090.429992] Lustre: server umount lustre-MDT0000 complete [ 3094.194979] LustreError: 94143:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769757717 with bad export cookie 14143219458515213834 [ 3094.203691] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3094.208918] LustreError: 94143:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3094.384346] LustreError: 96807:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3094.392503] LustreError: 96807:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3094.673503] Lustre: server umount lustre-MDT0001 complete [ 3104.261905] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 3106.468948] Key type lgssc unregistered [ 3106.657249] LNet: 97339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3106.671634] LNetError: 97339:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3106.684720] LNet: Removed LNI 192.168.206.151@tcp [ 3107.305540] Key type .llcrypt unregistered [ 3107.307085] Key type ._llcrypt unregistered [ 3119.373418] Key type ._llcrypt registered [ 3119.376625] Key type .llcrypt registered [ 3119.539548] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3120.893672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3120.916302] alg: No test for adler32 (adler32-zlib) [ 3121.993163] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 3122.291290] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 3123.944187] Key type lgssc registered [ 3124.937646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3130.996626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3131.004165] LustreError: 98381:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 3131.019255] LustreError: 98384:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 3131.028266] LustreError: 98381:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=4986 [ 3131.036124] LustreError: 98381:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3131.048988] LustreError: 98381:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3131.062891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3132.531174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3139.553094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3139.884531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3144.139317] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3148.689588] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing lsmod [ 3151.310025] LustreError: 99552:0:(tgt_mount.c:2535:server_fill_super()) cfs_race id 716 sleeping [ 3151.326322] LustreError: 99555:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 waking [ 3151.336809] LustreError: 99552:0:(tgt_mount.c:2535:server_fill_super()) cfs_fail_race id 716 awake: rc=4982 [ 3151.343857] LustreError: 99552:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3151.347748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3151.349050] LustreError: 99552:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3151.729231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3155.425847] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3155.441710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3155.459544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3156.975346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3156.989626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3157.000836] Lustre: Skipped 1 previous similar message [ 3160.379226] LustreError: 99739:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3160.462824] Lustre: server umount lustre-MDT0000 complete [ 3162.085243] LustreError: 98421:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3162.104480] LustreError: 98421:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3163.609031] LustreError: 98401:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769757786 with bad export cookie 1141661869507563858 [ 3163.616140] LustreError: 98401:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3163.617753] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3163.627939] Lustre: 98413:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.206.151@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3163.745528] LustreError: 99941:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3163.752329] LustreError: 99941:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3163.901905] Lustre: server umount lustre-MDT0001 complete [ 3177.737032] LustreError: 100142:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3177.743441] LustreError: 100142:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3178.109811] Lustre: server umount lustre-OST0000 complete [ 3186.312486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3186.793630] LustreError: 100620:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3186.871797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3191.217363] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3192.297218] LustreError: 100620:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3196.394164] LustreError: 100620:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3200.275907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3200.742576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3205.849944] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3210.991158] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3214.215476] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3221.056824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3221.325506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3227.003887] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3231.604778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3232.239854] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3240.941973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3240.949192] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3240.962792] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3240.969031] Lustre: Skipped 2 previous similar messages [ 3241.960052] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3244.617128] LustreError: 102861:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3244.627739] LustreError: 102861:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3244.807750] Lustre: server umount lustre-OST0000 complete [ 3252.194911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3252.204577] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3252.220675] Lustre: Skipped 1 previous similar message [ 3252.238094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3252.249145] Lustre: Skipped 3 previous similar messages [ 3254.946665] LustreError: 103062:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3254.950086] LustreError: 103062:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3255.303711] Lustre: server umount lustre-MDT0000 complete [ 3257.314121] LustreError: 100619:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3257.331600] LustreError: 100619:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3259.160549] LustreError: 103063:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769757882 with bad export cookie 1141661869507564523 [ 3259.163814] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3259.166710] LustreError: 103063:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3259.652368] Lustre: server umount lustre-MDT0001 complete [ 3270.544586] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 3273.640932] Key type lgssc unregistered [ 3273.991918] LNet: 103799:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3274.002419] LNetError: 103799:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3274.018899] LNet: Removed LNI 192.168.206.151@tcp [ 3275.223494] Key type .llcrypt unregistered [ 3275.225188] Key type ._llcrypt unregistered [ 3297.102710] Key type ._llcrypt registered [ 3297.105833] Key type .llcrypt registered [ 3297.235643] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3298.270664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3298.307518] alg: No test for adler32 (adler32-zlib) [ 3299.427214] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 3299.653425] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 3301.336360] Key type lgssc registered [ 3302.511540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3309.379052] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3309.407542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3311.197230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3315.816759] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3321.319369] LustreError: 105185:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3321.337769] LustreError: 105185:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3323.818771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3324.180543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3328.288559] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3333.645800] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3337.658577] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3344.842930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3345.295305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3351.731989] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3356.592947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3358.017932] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3363.682773] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3367.750740] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3372.008215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3372.011308] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3372.040289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3372.065348] Lustre: Skipped 1 previous similar message [ 3377.124436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3377.132743] Lustre: Skipped 1 previous similar message [ 3382.244059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3382.251507] Lustre: Skipped 1 previous similar message [ 3386.341640] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3386.395500] LustreError: 107533:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3386.486742] Lustre: server umount lustre-OST0000 complete [ 3390.945473] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3390.956592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3390.977867] Lustre: Skipped 1 previous similar message [ 3390.991718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3392.484966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3396.327405] LustreError: 107736:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3396.335550] LustreError: 107736:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3396.451859] Lustre: server umount lustre-MDT0000 complete [ 3397.601344] LustreError: 105182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3397.626566] LustreError: 105182:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3399.940466] LustreError: 105166:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758022 with bad export cookie 4850568244568057873 [ 3399.950355] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3399.952094] LustreError: 105166:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3400.073137] LustreError: 107938:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3400.082888] LustreError: 107938:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3400.273275] Lustre: server umount lustre-MDT0001 complete [ 3409.226802] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 3411.676888] Key type lgssc unregistered [ 3411.923110] LNet: 108470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3411.930245] LNetError: 108470:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3411.944692] LNet: Removed LNI 192.168.206.151@tcp [ 3412.663304] Key type .llcrypt unregistered [ 3412.665465] Key type ._llcrypt unregistered [ 3428.124233] Key type ._llcrypt registered [ 3428.127433] Key type .llcrypt registered [ 3428.274601] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3429.065672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3429.080506] alg: No test for adler32 (adler32-zlib) [ 3430.074320] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 3430.268344] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 3432.000159] Key type lgssc registered [ 3433.074218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3440.235763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3440.258254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3441.786403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3445.660374] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3451.362733] LustreError: 109674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3451.379651] LustreError: 109674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3454.539620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3454.902333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3459.035455] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3464.763452] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3469.282733] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3476.430949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3476.721404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3482.597729] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3487.046091] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3488.547498] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3497.442734] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3497.453421] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3497.459160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3497.463649] Lustre: Skipped 1 previous similar message [ 3497.480139] Lustre: Skipped 1 previous similar message [ 3502.563107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3503.171142] LustreError: 111916:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3503.308484] Lustre: server umount lustre-OST0000 complete [ 3507.682868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3507.697985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3507.699649] Lustre: Skipped 1 previous similar message [ 3507.718888] Lustre: Skipped 2 previous similar messages [ 3512.802260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3512.807397] Lustre: Skipped 1 previous similar message [ 3512.848712] LustreError: 112117:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3512.853554] LustreError: 112117:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3512.944703] Lustre: server umount lustre-MDT0000 complete [ 3516.337083] LustreError: 109656:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758139 with bad export cookie 14089393819666398574 [ 3516.350448] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3516.483474] LustreError: 112319:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3516.490499] LustreError: 112319:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3516.689066] Lustre: server umount lustre-MDT0001 complete [ 3527.538588] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 3529.882542] Key type lgssc unregistered [ 3530.146936] LNet: 112852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3530.157154] LNetError: 112852:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3530.191193] LNet: Removed LNI 192.168.206.151@tcp [ 3531.112669] Key type .llcrypt unregistered [ 3531.117542] Key type ._llcrypt unregistered [ 3552.031758] Key type ._llcrypt registered [ 3552.038478] Key type .llcrypt registered [ 3552.166794] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3553.243314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3553.299040] alg: No test for adler32 (adler32-zlib) [ 3554.314916] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 3554.512577] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 3556.208137] Key type lgssc registered [ 3557.171647] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3563.813544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3563.833948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3565.283034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3569.202246] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3575.781762] LustreError: 114228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3575.798688] LustreError: 114228:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3576.461821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3576.752202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3580.340179] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3585.091415] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3588.783719] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3594.420457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3594.757710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3600.213765] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3604.846474] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3610.105921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3614.585926] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3617.723363] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3617.727875] Lustre: Skipped 2 previous similar messages [ 3621.528080] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3621.531111] Lustre: Skipped 2 previous similar messages [ 3624.006569] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3624.033304] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3628.945087] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.51@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3628.953069] Lustre: Skipped 2 previous similar messages [ 3631.451957] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.51@tcp 192.168.0.[2,10]@tcp [ 3640.095336] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.51/24@tcp in log lustre-MDT0000 [ 3640.098793] Lustre: Skipped 2 previous similar messages [ 3647.043454] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.51/24@tcp [ 3647.049069] Lustre: Skipped 1 previous similar message [ 3654.114116] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3654.121291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3654.128676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3655.144398] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3655.145455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3655.149925] LustreError: Skipped 1 previous similar message [ 3655.172832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3655.251129] LustreError: 117872:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3655.396606] Lustre: server umount lustre-OST0000 complete [ 3658.721335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3658.727357] LustreError: Skipped 4 previous similar messages [ 3658.732724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3658.753342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3661.282760] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3661.283462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3661.300663] Lustre: Skipped 1 previous similar message [ 3664.563672] LustreError: 118072:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3664.567620] LustreError: 118072:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3664.695663] Lustre: server umount lustre-MDT0000 complete [ 3666.405880] LustreError: 114227:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3666.429163] LustreError: 114227:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3667.818149] LustreError: 117218:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758290 with bad export cookie 5755494257578963007 [ 3667.822900] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3667.824593] LustreError: 117218:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3667.986495] LustreError: 118274:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3667.991634] LustreError: 118274:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3668.091478] Lustre: server umount lustre-MDT0001 complete [ 3675.446590] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 3677.564359] Key type lgssc unregistered [ 3677.826480] LNet: 118805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3677.837129] LNetError: 118805:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3677.850964] LNet: Removed LNI 192.168.206.151@tcp [ 3678.463567] Key type .llcrypt unregistered [ 3678.466985] Key type ._llcrypt unregistered [ 3697.129994] Key type ._llcrypt registered [ 3697.132218] Key type .llcrypt registered [ 3697.199824] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3698.099117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3698.112381] alg: No test for adler32 (adler32-zlib) [ 3699.057572] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 3699.193376] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 3700.816186] Key type lgssc registered [ 3701.602467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3706.859742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3706.873104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3708.186281] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3708.204562] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.51/24@tcp [ 3708.228498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3711.139579] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3717.399729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3717.593380] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3717.610305] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.51/24@tcp [ 3717.660563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3720.460372] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3724.212400] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3727.044196] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3732.003144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3732.222336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3733.255797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3736.963497] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3740.671021] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3752.936715] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3752.938269] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3752.938608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3752.949863] Lustre: Skipped 1 previous similar message [ 3755.203363] LustreError: 122620:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3755.451801] Lustre: server umount lustre-OST0000 complete [ 3758.562124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3758.579978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3758.590977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3758.596296] Lustre: Skipped 1 previous similar message [ 3763.169815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3763.169903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3763.189683] Lustre: Skipped 1 previous similar message [ 3764.390982] LustreError: 122821:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3764.395772] LustreError: 122821:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3764.642674] Lustre: server umount lustre-MDT0000 complete [ 3767.911829] LustreError: 120304:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758390 with bad export cookie 1192627577429744550 [ 3767.916686] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3767.919641] LustreError: 120304:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3768.064622] LustreError: 123023:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3768.070615] LustreError: 123023:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3768.238624] Lustre: server umount lustre-MDT0001 complete [ 3775.739929] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 3777.584943] Key type lgssc unregistered [ 3777.823218] LNet: 123553:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3777.831535] LNetError: 123553:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3778.863695] LNet: Removed LNI 192.168.206.151@tcp [ 3779.427635] Key type .llcrypt unregistered [ 3779.429873] Key type ._llcrypt unregistered [ 3798.818337] Key type ._llcrypt registered [ 3798.819915] Key type .llcrypt registered [ 3798.895309] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 3809.266145] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 3810.025916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3810.044865] alg: No test for adler32 (adler32-zlib) [ 3811.144464] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 3811.322675] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 3812.984267] Key type lgssc registered [ 3813.832479] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3838.082480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3838.100532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3839.265932] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3839.284284] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3839.338578] Lustre: lustre-MDT0000: new disk, initializing [ 3839.388370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3839.410879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3842.190293] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3849.569864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3849.639783] Lustre: 126892:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3849.686805] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3849.689784] Lustre: Skipped 1 previous similar message [ 3849.738546] Lustre: lustre-MDT0001: new disk, initializing [ 3849.809413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3849.855901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3849.870641] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3852.721715] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3858.671844] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3861.438458] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3866.179335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3866.360893] Lustre: lustre-OST0000: new disk, initializing [ 3866.363748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3866.443866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3870.429294] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3873.837284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3873.846991] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3873.912714] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3875.903364] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3879.148901] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3879.296517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3882.191315] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3882.314762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3884.007295] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3884.009100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3884.015405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3884.015818] Lustre: Skipped 1 previous similar message [ 3889.123359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3889.126239] Lustre: Skipped 2 previous similar messages [ 3889.657419] LustreError: 129823:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3889.794762] Lustre: server umount lustre-OST0000 complete [ 3894.245083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3894.253385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3894.267881] Lustre: Skipped 1 previous similar message [ 3898.647957] LustreError: 130025:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3898.655045] LustreError: 130025:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3898.720723] Lustre: server umount lustre-MDT0000 complete [ 3901.331440] LustreError: 126887:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758524 with bad export cookie 5566475458754931819 [ 3901.337345] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3901.346918] LustreError: 126887:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3901.456665] LustreError: 130227:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3901.466652] LustreError: 130227:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3901.602558] Lustre: server umount lustre-MDT0001 complete [ 3908.354291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3908.643324] LustreError: 130703:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3908.651974] LustreError: 130703:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3908.696636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3911.436156] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3913.699377] LustreError: 130703:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3917.526826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3917.834410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3921.141419] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3925.446879] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3928.505837] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3933.450870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3933.682186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3938.207380] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3942.474781] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3945.787822] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3953.369124] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 3956.778557] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3956.946489] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3963.340443] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3963.479780] Lustre: lustre-OST0001: new disk, initializing [ 3963.485214] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3963.567651] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3967.911855] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 3973.731235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3973.735595] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3973.799174] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3973.826194] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3973.956775] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3976.740740] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3976.878346] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3979.337278] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ee74b48e000.ost_server_uuid 50 [ 3980.439105] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8ee74b48e000.ost_server_uuid in FULL state after 0 sec [ 3989.477748] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3989.485150] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3989.491340] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3989.498629] Lustre: Skipped 1 previous similar message [ 3989.507294] Lustre: Skipped 2 previous similar messages [ 3992.444252] LustreError: 134991:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3992.450940] LustreError: 134991:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3992.507786] Lustre: server umount lustre-OST0001 complete [ 3994.596036] LustreError: 132295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3994.609872] LustreError: 132295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3999.209416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3999.210691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3999.218014] LustreError: Skipped 2 previous similar messages [ 3999.226108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3999.718162] LustreError: 132296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3999.727292] LustreError: 132296:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4001.286198] LustreError: 135192:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4001.296535] LustreError: 135192:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4001.378531] Lustre: server umount lustre-OST0000 complete [ 4004.839358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4004.841086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4004.850568] Lustre: Skipped 1 previous similar message [ 4009.955782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4009.966063] Lustre: Skipped 4 previous similar messages [ 4010.494708] LustreError: 135395:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4010.500570] LustreError: 135395:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4010.642918] Lustre: server umount lustre-MDT0000 complete [ 4013.313334] LustreError: 130683:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758636 with bad export cookie 5566475458754932701 [ 4013.315454] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4013.323421] LustreError: 130683:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4013.582986] Lustre: server umount lustre-MDT0001 complete [ 4027.404633] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 4028.999632] Key type lgssc unregistered [ 4029.195426] LNet: 136584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4029.200500] LNetError: 136584:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4029.220960] LNet: Removed LNI 192.168.206.151@tcp [ 4029.765135] Key type .llcrypt unregistered [ 4029.770124] Key type ._llcrypt unregistered [ 4052.038054] Key type ._llcrypt registered [ 4052.039418] Key type .llcrypt registered [ 4052.084415] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 4060.917873] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 4061.473836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4061.499457] alg: No test for adler32 (adler32-zlib) [ 4062.437968] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 4062.569185] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 4064.168161] Key type lgssc registered [ 4064.896767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4085.184167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4085.193848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4086.317653] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4086.339813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4086.382993] Lustre: lustre-MDT0000: new disk, initializing [ 4086.425299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4086.435196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4088.674397] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4094.705310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4094.780800] Lustre: 140675:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4094.822637] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4094.824840] Lustre: Skipped 1 previous similar message [ 4094.882624] Lustre: lustre-MDT0001: new disk, initializing [ 4094.933823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4094.950128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4094.954734] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4097.242967] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4101.401362] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4103.512380] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4107.026314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4107.159434] Lustre: lustre-OST0000: new disk, initializing [ 4107.164838] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4107.217437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4108.360344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4108.366116] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4108.437758] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4110.015148] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4114.129109] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4117.723659] Lustre: Failing over lustre-OST0000 [ 4117.757074] LustreError: 143110:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4117.820276] Lustre: server umount lustre-OST0000 complete [ 4132.863650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4133.076680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4133.093976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4133.103354] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4134.112178] Lustre: 138901:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769758741/real 1769758741] req@0000000070890421 x1855726482910464/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1769758757 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4136.564947] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4137.356401] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4137.953210] Lustre: 138902:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769758744/real 1769758744] req@00000000486b3e33 x1855726482911616/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1769758760 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4137.968913] Lustre: 138902:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4138.499366] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4138.499801] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4138.647184] Lustre: Failing over lustre-MDT0000 [ 4138.770218] LustreError: 144083:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4138.773550] LustreError: 144083:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4138.857665] Lustre: server umount lustre-MDT0000 complete [ 4140.704138] Lustre: 138903:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769758747/real 1769758747] req@000000008a2993e6 x1855726482912384/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1769758763 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4141.026158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4141.035872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4141.043761] Lustre: Skipped 1 previous similar message [ 4143.589627] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4145.632786] Lustre: 138901:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769758752/real 1769758752] req@0000000083ef8ae9 x1855726482913536/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1769758768 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4147.567297] LustreError: 143225:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4147.579829] LustreError: 143225:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4148.708534] LustreError: 144201:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4148.724159] LustreError: 144201:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4152.683331] LustreError: 140683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4153.822803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4153.913163] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4153.917449] Lustre: 140679:0:(mgc_request.c:1911:mgc_process_log()) MGC192.168.206.151@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4154.038080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4154.061956] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4156.316706] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4157.802820] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4159.484606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4159.489048] Lustre: Skipped 1 previous similar message [ 4159.499106] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4159.532138] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4164.066689] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4164.072314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4164.078081] Lustre: Skipped 1 previous similar message [ 4164.081045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4164.587992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4168.189081] LustreError: 144969:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4168.191273] LustreError: 144969:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4168.262769] Lustre: server umount lustre-OST0000 complete [ 4174.817399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4174.817992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4174.818843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4174.822252] Lustre: Skipped 2 previous similar messages [ 4176.464901] LustreError: 145174:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4176.467625] LustreError: 145174:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4176.514124] Lustre: server umount lustre-MDT0000 complete [ 4178.585236] LustreError: 140667:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769758801 with bad export cookie 2973154584760609830 [ 4178.589236] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4178.599248] LustreError: 140667:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4178.842833] Lustre: server umount lustre-MDT0001 complete [ 4185.067353] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 4186.819706] Key type lgssc unregistered [ 4187.009368] LNet: 145908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4187.013697] LNetError: 145908:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4188.076636] LNet: Removed LNI 192.168.206.151@tcp [ 4188.603580] Key type .llcrypt unregistered [ 4188.605246] Key type ._llcrypt unregistered [ 4215.994858] Key type ._llcrypt registered [ 4215.996853] Key type .llcrypt registered [ 4216.061838] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 4216.849570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4216.913746] alg: No test for adler32 (adler32-zlib) [ 4217.913565] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 4218.056266] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 4219.683596] Key type lgssc registered [ 4220.377690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4225.315040] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4225.337039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4226.675467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4229.258393] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4234.476742] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4236.259923] LustreError: 147408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4236.269369] LustreError: 147408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4237.253419] LustreError: 147436:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4237.261445] LustreError: 147436:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4237.266321] LustreError: 147436:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 11, retries 0, failed: rc = -5 [ 4237.359596] LustreError: 147958:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4237.479711] Lustre: server umount lustre-MDT0000 complete [ 4246.732615] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 4253.528351] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 4275.252627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4275.405969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4275.434775] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4275.497953] Lustre: lustre-MDT0000: new disk, initializing [ 4275.560135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4275.578309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4277.978211] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4284.589166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4284.666146] Lustre: 150589:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4284.705357] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4284.708918] Lustre: Skipped 1 previous similar message [ 4284.762911] Lustre: lustre-MDT0001: new disk, initializing [ 4284.818377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4284.845657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4284.861586] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4288.100132] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4292.592822] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4294.651108] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4298.881485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4299.064698] Lustre: lustre-OST0000: new disk, initializing [ 4299.068178] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4299.145676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4301.794588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4301.807665] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4301.878369] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4302.564964] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4306.988773] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4309.548555] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4310.513317] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:42:13 (1769758933) [ 4479.458641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4479.462283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4479.470379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4480.992921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4480.993483] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4480.996558] Lustre: Skipped 1 previous similar message [ 4481.012034] Lustre: Skipped 1 previous similar message [ 4483.786474] LustreError: 154094:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4483.788673] LustreError: 154094:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4483.837432] Lustre: server umount lustre-MDT0000 complete [ 4485.814785] LustreError: 150581:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759108 with bad export cookie 86017522184168904 [ 4485.815603] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4485.819473] LustreError: 150581:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4486.114299] LustreError: 150594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4486.114465] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4486.142384] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4486.145067] Lustre: Skipped 1 previous similar message [ 4490.145265] LustreError: 150594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4490.149099] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4491.974800] LustreError: 154297:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4491.980470] LustreError: 154297:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4492.063433] Lustre: server umount lustre-MDT0001 complete [ 4500.716794] LustreError: 154499:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4500.720481] LustreError: 154499:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4500.793491] Lustre: server umount lustre-OST0000 complete [ 4506.908515] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 4513.466193] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 4537.621927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4537.778093] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4537.802921] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4537.851815] Lustre: lustre-MDT0000: new disk, initializing [ 4537.914531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4537.925706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4540.509801] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4546.796985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4546.867276] Lustre: 156828:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4546.905281] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4546.908791] Lustre: Skipped 1 previous similar message [ 4546.953710] Lustre: lustre-MDT0001: new disk, initializing [ 4547.018797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4547.039886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4547.047525] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4549.364947] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4553.959836] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4556.001597] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4559.622149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4559.749975] Lustre: lustre-OST0000: new disk, initializing [ 4559.754709] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4562.563815] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4567.218882] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4568.095636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4568.106611] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4568.196799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4569.904649] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4570.042994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4572.072484] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4572.177609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4578.281082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4578.293053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4578.308881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4578.316700] Lustre: Skipped 1 previous similar message [ 4579.324907] LustreError: 159753:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4579.332889] LustreError: 159753:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4579.396718] Lustre: server umount lustre-OST0000 complete [ 4582.880876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4582.883884] LustreError: Skipped 1 previous similar message [ 4582.886114] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4582.894927] Lustre: Skipped 1 previous similar message [ 4587.483582] LustreError: 159954:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4587.488919] LustreError: 159954:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4587.582963] Lustre: server umount lustre-MDT0000 complete [ 4588.514419] LustreError: 156837:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4588.521120] LustreError: 156837:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4589.764629] LustreError: 156819:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759212 with bad export cookie 86017522184327734 [ 4589.766241] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4589.769779] LustreError: 156819:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4589.934265] Lustre: server umount lustre-MDT0001 complete [ 4595.032718] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:46:57 (1769759217) [ 4603.893318] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 4608.959763] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 4626.742818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4626.891560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4626.910885] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4626.951071] Lustre: lustre-MDT0000: new disk, initializing [ 4626.993720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4626.996622] Lustre: Skipped 1 previous similar message [ 4627.006506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4629.303482] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4635.128657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4635.198709] Lustre: 163133:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4635.333108] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4637.284286] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4641.437862] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4643.120964] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4646.381942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4648.513634] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4648.559192] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4649.027107] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4653.234920] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4658.659837] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4658.660304] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4658.667435] Lustre: Skipped 1 previous similar message [ 4658.671843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4658.675407] Lustre: Skipped 4 previous similar messages [ 4662.641079] LustreError: 165671:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4662.644165] LustreError: 165671:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4662.695874] Lustre: server umount lustre-OST0000 complete [ 4666.339091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4670.631509] Lustre: server umount lustre-MDT0000 complete [ 4672.247603] LustreError: 163123:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759295 with bad export cookie 86017522184328602 [ 4672.253802] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4672.416597] Lustre: server umount lustre-MDT0001 complete [ 4677.084713] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:48:19 (1769759299) [ 4684.924176] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 4690.093165] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 4707.406147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4707.570910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4707.589839] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4707.591786] Lustre: Skipped 3 previous similar messages [ 4707.631854] Lustre: lustre-MDT0000: new disk, initializing [ 4707.634107] Lustre: Skipped 2 previous similar messages [ 4707.684064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4707.687490] Lustre: Skipped 2 previous similar messages [ 4707.695882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4707.702034] Lustre: Skipped 2 previous similar messages [ 4709.520672] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4714.993790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4715.042079] Lustre: 169048:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4715.185318] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4716.897865] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4720.882090] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4722.547590] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4725.619981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4728.589311] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4732.706866] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4732.970934] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4743.137585] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4743.138820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4743.142381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4743.145878] Lustre: Skipped 3 previous similar messages [ 4743.148036] Lustre: Skipped 3 previous similar messages [ 4746.998106] LustreError: 171587:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4747.004269] LustreError: 171587:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 4747.049948] Lustre: server umount lustre-OST0000 complete [ 4755.080083] Lustre: server umount lustre-MDT0000 complete [ 4756.874847] LustreError: 169040:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759379 with bad export cookie 86017522184329687 [ 4756.876450] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4756.880206] LustreError: 169040:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4757.071555] Lustre: server umount lustre-MDT0001 complete [ 4761.621494] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 4762.956933] Key type lgssc unregistered [ 4763.109527] LNet: 172521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4763.112537] LNetError: 172521:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4763.122375] LNet: Removed LNI 192.168.206.151@tcp [ 4763.519563] Key type .llcrypt unregistered [ 4763.520832] Key type ._llcrypt unregistered [ 4774.616854] Key type ._llcrypt registered [ 4774.618558] Key type .llcrypt registered [ 4774.667266] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 4775.038042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4775.076647] alg: No test for adler32 (adler32-zlib) [ 4775.964402] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 4776.073982] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 4777.673166] Key type lgssc registered [ 4778.155147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4781.313835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4781.322056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4782.547119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4784.123372] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4787.590358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4787.770786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4789.605792] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4792.294637] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4794.127430] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4797.500368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4797.673913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4800.406606] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4802.967943] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4808.941306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4829.667276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4829.670090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4829.676311] LustreError: Skipped 1 previous similar message [ 4829.678773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4829.684868] Lustre: Skipped 1 previous similar message [ 4829.696580] Lustre: Skipped 1 previous similar message [ 4832.499174] LustreError: 176156:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4832.550564] Lustre: server umount lustre-OST0000 complete [ 4834.785641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4834.787623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4834.789380] Lustre: Skipped 1 previous similar message [ 4834.797414] Lustre: Skipped 1 previous similar message [ 4839.905177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4839.909389] Lustre: Skipped 1 previous similar message [ 4845.026038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4845.028623] Lustre: Skipped 1 previous similar message [ 4849.120694] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4849.122786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4849.129791] Lustre: Skipped 1 previous similar message [ 4849.171568] LustreError: 176358:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4849.175283] LustreError: 176358:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4849.209139] Lustre: server umount lustre-MDT0000 complete [ 4850.967313] LustreError: 173875:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759473 with bad export cookie 7195893015487323449 [ 4850.972602] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4850.973660] LustreError: 173875:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4851.054202] LustreError: 176562:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4851.056304] LustreError: 176562:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4851.127416] Lustre: server umount lustre-MDT0001 complete [ 4855.870560] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 4857.220710] Key type lgssc unregistered [ 4857.380072] LNet: 177094:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4857.383803] LNetError: 177094:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4857.392344] LNet: Removed LNI 192.168.206.151@tcp [ 4857.753084] Key type .llcrypt unregistered [ 4857.754768] Key type ._llcrypt unregistered [ 4868.893946] Key type ._llcrypt registered [ 4868.895107] Key type .llcrypt registered [ 4868.936199] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 4869.280122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4869.310107] alg: No test for adler32 (adler32-zlib) [ 4870.189106] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 4870.274112] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 4871.864294] Key type lgssc registered [ 4872.283997] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4875.273953] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4875.281185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4876.483237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4878.056728] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4881.803118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4882.000939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4883.791637] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4886.327707] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4887.851892] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4891.136879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4891.237461] LustreError: 180061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4891.249887] LustreError: 180061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4891.321882] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4894.568448] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4897.317551] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4901.865882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4906.979744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4906.980320] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4906.985443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4906.986641] LustreError: Skipped 1 previous similar message [ 4908.921974] Lustre: lustre-OST0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 4908.925243] Lustre: Skipped 1 previous similar message [ 4912.098340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4912.178260] LustreError: 180708:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4912.224990] Lustre: server umount lustre-OST0000 complete [ 4914.875882] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8ee749f7d800.ost_server_uuid 50 [ 4915.703749] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4927.457156] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4927.459777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4927.462025] Lustre: Skipped 2 previous similar messages [ 4927.464591] Lustre: Skipped 1 previous similar message [ 4932.578338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4932.580980] Lustre: Skipped 2 previous similar messages [ 4933.250818] LustreError: 181073:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4933.254668] LustreError: 181073:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4933.295241] Lustre: server umount lustre-MDT0000 complete [ 4935.115210] LustreError: 178448:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759558 with bad export cookie 18148186006022861897 [ 4935.118232] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4935.118961] LustreError: 178448:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4935.195897] LustreError: 181275:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4935.198495] LustreError: 181275:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4935.287796] Lustre: server umount lustre-MDT0001 complete [ 4939.746665] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:52:42 (1769759562) [ 4943.174655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4943.427835] LustreError: 181945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4943.479817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4945.544461] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4948.962382] LustreError: 181945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4950.455608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4950.722082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4953.249207] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4956.594975] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4959.050886] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4962.548886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4962.745374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4966.195604] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4968.884932] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4968.963356] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4973.009340] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4973.111651] Lustre: lustre-OST0001: new disk, initializing [ 4973.113901] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4976.171442] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 4977.705432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4977.709710] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4977.772887] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4981.108938] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4982.758893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4982.763345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4982.770834] Lustre: Skipped 1 previous similar message [ 4982.779184] Lustre: Skipped 1 previous similar message [ 4988.595235] LustreError: 185139:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4988.597868] LustreError: 185139:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4988.623829] Lustre: server umount lustre-OST0000 complete [ 4991.740397] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4991.840293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4991.861697] LustreError: 183533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4993.823181] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4993.941291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4996.969428] LustreError: 183533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4996.977191] LustreError: 183533:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5008.355334] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5008.359485] LustreError: 184652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5008.359856] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5008.360578] LustreError: Skipped 1 previous similar message [ 5008.362338] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5008.366660] LustreError: 184652:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 5008.376791] Lustre: Skipped 4 previous similar messages [ 5010.543158] LustreError: 185842:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5010.551061] LustreError: 185842:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5010.595567] Lustre: server umount lustre-OST0001 complete [ 5012.450082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5012.455521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5012.463888] Lustre: Skipped 1 previous similar message [ 5018.587672] LustreError: 186044:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5018.590344] LustreError: 186044:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5018.662522] Lustre: server umount lustre-MDT0000 complete [ 5020.719569] LustreError: 181924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759643 with bad export cookie 18148186006022862954 [ 5020.724037] LustreError: 181924:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5020.725874] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5020.902181] Lustre: server umount lustre-MDT0001 complete [ 5033.071184] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:54:15 (1769759655) [ 5036.271840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5036.460721] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5036.480814] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5036.585359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5036.587684] Lustre: Skipped 1 previous similar message [ 5038.230671] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5042.512939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5042.573980] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5042.606685] Lustre: 187712:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5044.867218] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5047.494891] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5049.340941] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5053.284423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5053.401813] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5056.420239] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5059.354923] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5063.227823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5063.320484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5063.364978] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5066.678362] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5069.818354] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5071.363545] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5071.366114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5071.372303] Lustre: Skipped 2 previous similar messages [ 5071.384359] Lustre: Skipped 6 previous similar messages [ 5077.365618] LustreError: 190885:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5077.373120] LustreError: 190885:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5077.433252] Lustre: server umount lustre-OST0000 complete [ 5080.438325] LustreError: 189334:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5091.811990] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5091.816924] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5091.818734] LustreError: Skipped 1 previous similar message [ 5091.828417] Lustre: Skipped 1 previous similar message [ 5095.272834] Lustre: server umount lustre-OST0001 complete [ 5098.981373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5103.541406] Lustre: server umount lustre-MDT0000 complete [ 5105.766483] LustreError: 187705:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759728 with bad export cookie 18148186006022864340 [ 5105.773497] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5105.776810] LustreError: 187705:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5105.978697] Lustre: server umount lustre-MDT0001 complete [ 5118.713219] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:55:41 (1769759741) [ 5126.923254] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 5131.255362] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 5149.162637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5149.295732] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5149.310252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5149.352719] Lustre: lustre-MDT0000: new disk, initializing [ 5149.413774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5149.417681] Lustre: Skipped 3 previous similar messages [ 5149.425756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5151.429613] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5157.438435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5157.504193] Lustre: 195289:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5157.535543] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5157.538285] Lustre: Skipped 1 previous similar message [ 5157.591273] Lustre: lustre-MDT0001: new disk, initializing [ 5157.696326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5157.708258] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5159.561681] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5163.403831] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5165.110641] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5168.162284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5168.315930] Lustre: lustre-OST0000: new disk, initializing [ 5168.319549] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5170.885824] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5171.464343] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5171.467838] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5171.502331] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5174.854214] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5176.764360] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5176.876168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5179.073333] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5179.191434] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5181.922372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5181.924307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5181.926274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5181.929963] Lustre: Skipped 3 previous similar messages [ 5181.941536] Lustre: Skipped 9 previous similar messages [ 5186.290499] LustreError: 198223:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5186.293358] LustreError: 198223:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 5186.362612] Lustre: server umount lustre-OST0000 complete [ 5194.152705] Lustre: server umount lustre-MDT0000 complete [ 5195.915293] LustreError: 195281:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759818 with bad export cookie 18148186006022865859 [ 5195.918187] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5195.920436] LustreError: 195281:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5196.079988] Lustre: server umount lustre-MDT0001 complete [ 5200.381057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5200.681800] LustreError: 199100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5200.696466] LustreError: 199100:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5202.704467] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5207.020047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5209.059468] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5211.388887] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5212.999598] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5216.208897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5216.382423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5216.385257] Lustre: Skipped 4 previous similar messages [ 5219.028990] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5221.422352] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5223.092322] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5226.274396] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5227.985534] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5228.091022] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5231.079132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5231.082395] LustreError: Skipped 2 previous similar messages [ 5235.053622] Lustre: server umount lustre-OST0000 complete [ 5237.747541] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5239.600105] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5263.727589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5266.353937] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5268.648320] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5269.368196] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5276.907438] LustreError: 203485:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5276.909517] LustreError: 203485:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 5276.950423] Lustre: server umount lustre-OST0000 complete [ 5279.201165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5279.205375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5279.212691] Lustre: Skipped 4 previous similar messages [ 5285.031983] Lustre: server umount lustre-MDT0000 complete [ 5286.900316] LustreError: 199079:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769759909 with bad export cookie 18148186006022866741 [ 5286.904551] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5286.907407] LustreError: 199079:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5287.066603] Lustre: server umount lustre-MDT0001 complete [ 5290.943746] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:58:34 (1769759914) [ 5293.605516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5293.797123] LustreError: 204565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5293.803417] LustreError: 204565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5295.326071] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5298.544046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5300.191164] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5302.309518] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5303.794568] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5306.459985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5308.893979] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5311.034842] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5312.558858] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5319.807668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 7 sec [ 5321.291623] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5321.386678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5324.151322] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5324.248953] Lustre: lustre-OST0001: new disk, initializing [ 5324.251942] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5326.529862] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5330.040758] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5330.958794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5330.962730] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5330.994952] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5331.618271] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5331.707866] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5333.325159] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5333.410218] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5336.038268] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5336.040990] Lustre: Skipped 13 previous similar messages [ 5340.236143] Lustre: server umount lustre-OST0001 complete [ 5342.636342] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5342.745332] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5344.345594] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5344.427981] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5368.633399] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5368.758637] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5368.762696] Lustre: Skipped 5 previous similar messages [ 5371.258965] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5373.487881] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5373.928705] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5373.937825] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5373.938213] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5373.945616] Lustre: Skipped 1 previous similar message [ 5374.194867] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5377.086922] Lustre: server umount lustre-OST0001 complete [ 5379.041557] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5379.045272] LustreError: Skipped 2 previous similar messages [ 5395.549202] Lustre: server umount lustre-OST0000 complete [ 5403.367904] Lustre: server umount lustre-MDT0000 complete [ 5404.872331] LustreError: 204544:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760027 with bad export cookie 18148186006022867672 [ 5404.876122] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5404.876404] LustreError: 204544:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5411.006669] LustreError: 211081:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5411.008974] LustreError: 211081:0:(obd_class.h:479:obd_check_dev()) Skipped 37 previous similar messages [ 5411.071593] Lustre: server umount lustre-MDT0001 complete [ 5420.549704] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 03:00:43 (1769760043) [ 5423.049366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5423.154540] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5423.173845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5424.773774] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5428.222689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5428.280247] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5428.292226] Lustre: 212549:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5429.979666] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5432.144499] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5433.617633] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5436.412038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5436.494351] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5439.215461] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5441.564628] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5446.076953] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5446.141787] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5448.534647] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5450.838206] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5451.552178] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5453.529324] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5456.711741] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5458.389263] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5458.494038] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5460.192784] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ee74a9ba000.ost_server_uuid 50 [ 5460.919949] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8ee74a9ba000.ost_server_uuid in FULL state after 0 sec [ 5461.390576] Lustre: Permanently deactivating lustre-OST0001 [ 5461.399585] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5461.768051] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 5462.737644] Lustre: Permanently reactivating lustre-OST0001 [ 5470.213737] Lustre: server umount lustre-OST0001 complete [ 5471.218408] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5471.225688] Lustre: Skipped 9 previous similar messages [ 5471.229047] LustreError: 214557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5471.236342] LustreError: 214557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 27 previous similar messages [ 5477.983131] Lustre: server umount lustre-OST0000 complete [ 5489.671640] Lustre: server umount lustre-MDT0000 complete [ 5491.313061] LustreError: 212541:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760114 with bad export cookie 18148186006022868946 [ 5491.319410] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5491.680143] LustreError: 216689:0:(osp_dev.c:692:osp_process_config()) lustre-OST0001-osc-MDT0001: unknown param osc.active=1 [ 5491.687828] LustreError: 212552:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 5491.848365] Lustre: server umount lustre-MDT0001 complete [ 5501.825952] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:02:04 (1769760124) [ 5504.973529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5505.100938] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5505.116674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5505.122653] Lustre: Skipped 5 previous similar messages [ 5506.674733] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5509.941867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5509.984820] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5509.997153] Lustre: 218693:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5511.514591] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5513.427986] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5514.731048] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5517.313161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5517.397218] Lustre: Permanently deactivating lustre-OST0000 [ 5517.403097] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5517.890314] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5517.893731] Lustre: Skipped 1 previous similar message [ 5517.895478] LustreError: 220415:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 5517.900879] LustreError: 220415:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 5519.624994] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5521.660789] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5524.600811] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5527.259286] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5529.550304] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5531.683070] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5535.706535] Lustre: Permanently reactivating lustre-OST0000 [ 5535.711812] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5535.714063] Lustre: Skipped 2 previous similar messages [ 5540.380718] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 5540.395252] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5540.411352] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5540.443103] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 5540.456194] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5544.937512] LustreError: lustre-OST0001-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 5544.941710] LustreError: Skipped 5 previous similar messages [ 5548.381987] Lustre: server umount lustre-OST0001 complete [ 5564.384371] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5564.449465] Lustre: server umount lustre-OST0000 complete [ 5572.329642] Lustre: server umount lustre-MDT0000 complete [ 5573.775526] LustreError: 218684:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760196 with bad export cookie 18148186006022870661 [ 5573.779579] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5573.781079] LustreError: 218684:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5573.947254] Lustre: server umount lustre-MDT0001 complete [ 5578.621938] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 5579.859530] Key type lgssc unregistered [ 5580.024776] LNet: 223076:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5580.029807] LNetError: 223076:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5580.037689] LNet: Removed LNI 192.168.206.151@tcp [ 5580.430090] Key type .llcrypt unregistered [ 5580.431936] Key type ._llcrypt unregistered [ 5591.342499] Key type ._llcrypt registered [ 5591.343826] Key type .llcrypt registered [ 5591.387896] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 5591.719508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5591.727884] alg: No test for adler32 (adler32-zlib) [ 5592.610260] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 5592.701789] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 5594.304169] Key type lgssc registered [ 5594.750700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5598.894183] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5598.901203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5600.080768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5601.454810] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5604.701200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5604.755422] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5604.770722] Lustre: 224487:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5604.777038] Lustre: Permanently deactivating lustre-MDT0001 [ 5604.786135] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5604.919862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5606.508586] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5607.347423] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5607.350829] LustreError: 224529:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5607.356520] LustreError: 224529:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5607.361420] LustreError: 224529:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 1, failed: rc = -108 [ 5608.571446] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5609.977307] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5612.528495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5612.660917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5614.624596] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5614.693929] LustreError: 226115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5614.702157] LustreError: 226115:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5616.606667] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5619.507967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5619.617167] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5621.917324] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5624.201336] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5624.809153] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5625.832185] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5625.833023] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5631.405191] Lustre: Permanently reactivating lustre-MDT0001 [ 5631.414132] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5631.416898] Lustre: Skipped 2 previous similar messages [ 5631.931206] Lustre: lustre-MDT0001: Client 47d5ce30-954d-4d7a-a130-c68b8d910284 (at 192.168.206.51@tcp) reconnecting [ 5634.075848] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5634.758610] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5637.014473] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5637.100509] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5638.653161] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5641.793268] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 3 sec [ 5642.825081] Lustre: Permanently deactivating lustre-MDT0001 [ 5642.840245] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5642.844825] Lustre: Skipped 2 previous similar messages [ 5646.222292] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5656.033237] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5656.037401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5656.043808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5660.642649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5660.643408] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5660.645402] Lustre: Skipped 2 previous similar messages [ 5660.649285] Lustre: Skipped 2 previous similar messages [ 5661.888801] LustreError: 228598:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 5661.920762] Lustre: server umount lustre-MDT0000 complete [ 5663.366422] LustreError: 225220:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760286 with bad export cookie 16648179602755649568 [ 5663.368178] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5663.370070] LustreError: 225220:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5663.452209] LustreError: 228799:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 5663.454818] LustreError: 228799:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5663.532974] Lustre: server umount lustre-MDT0001 complete [ 5671.093076] LustreError: 229000:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 5671.095537] LustreError: 229000:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5671.126866] Lustre: server umount lustre-OST0000 complete [ 5675.823874] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:04:58 (1769760298) [ 5680.632507] LustreError: 230007:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5680.635193] LustreError: 230007:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5680.685426] Lustre: server umount lustre-OST0001 complete [ 5683.323471] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 5687.209316] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 5702.417436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5702.568772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5702.575608] Lustre: Skipped 2 previous similar messages [ 5702.594790] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5702.646235] Lustre: lustre-MDT0000: new disk, initializing [ 5702.692067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5702.699873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5704.527584] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5709.618087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5709.665123] Lustre: 232178:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5709.691070] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5709.692934] Lustre: Skipped 1 previous similar message [ 5709.733492] Lustre: lustre-MDT0001: new disk, initializing [ 5709.785482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5709.789505] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5711.226877] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5714.848383] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5716.350425] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5718.995629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5719.104728] Lustre: lustre-OST0000: new disk, initializing [ 5719.108300] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5719.146477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5719.149179] Lustre: Skipped 1 previous similar message [ 5721.454419] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5724.868164] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5728.765728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5728.770044] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5728.802543] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5730.775866] LustreError: 232183:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5735.308891] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5735.376187] Lustre: lustre-OST0001: new disk, initializing [ 5735.378468] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5735.411221] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5737.460637] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5740.822479] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5740.874433] LustreError: 232183:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5740.885575] LustreError: 232183:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 5741.462561] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5745.167792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5745.172646] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5745.203409] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5750.976258] LustreError: 232183:0:(mdt_handler.c:3252:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 5755.249319] Lustre: lustre-OST0001: Not available for connect from 192.168.206.51@tcp (stopping) [ 5755.251659] Lustre: Skipped 1 previous similar message [ 5755.364632] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5755.367897] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5755.372956] Lustre: Skipped 1 previous similar message [ 5757.745204] LustreError: 235663:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5757.748220] LustreError: 235663:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5757.773327] Lustre: server umount lustre-OST0001 complete [ 5760.487356] LustreError: 233902:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5760.499493] LustreError: 233902:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5765.604972] LustreError: 234522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5765.612039] LustreError: 234522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5770.722707] LustreError: 233903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5770.723287] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5770.724317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5770.724324] Lustre: Skipped 2 previous similar messages [ 5770.727986] LustreError: 233903:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5770.738880] Lustre: Skipped 1 previous similar message [ 5775.845143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5775.845950] LustreError: 234522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5775.847436] Lustre: Skipped 2 previous similar messages [ 5775.856248] LustreError: 234522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5776.109174] LustreError: 235886:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5776.111261] LustreError: 235886:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5776.152979] Lustre: server umount lustre-OST0000 complete [ 5780.961856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5780.966763] Lustre: Skipped 1 previous similar message [ 5783.774359] Lustre: server umount lustre-MDT0000 complete [ 5785.159718] LustreError: 233917:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760408 with bad export cookie 16648179602755652158 [ 5785.160650] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5785.166817] LustreError: 233917:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5791.361757] Lustre: server umount lustre-MDT0001 complete [ 5795.151163] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 5796.267602] Key type lgssc unregistered [ 5796.403433] LNet: 236821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5796.406511] LNetError: 236821:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5796.414569] LNet: Removed LNI 192.168.206.151@tcp [ 5796.749205] Key type .llcrypt unregistered [ 5796.750986] Key type ._llcrypt unregistered [ 5812.132861] Key type ._llcrypt registered [ 5812.134584] Key type .llcrypt registered [ 5812.183912] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 5812.674810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5812.694625] alg: No test for adler32 (adler32-zlib) [ 5813.562366] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 5813.663117] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 5815.248982] Key type lgssc registered [ 5815.740640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5818.703183] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5818.712722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5819.834472] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5819.849177] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5819.924837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5821.450022] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5825.085891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5825.137341] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5825.152105] Lustre: 238986:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5825.268986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5826.880562] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5828.985821] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5830.430474] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5833.038197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5833.121055] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5833.211164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5835.419418] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5836.235024] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 5837.714556] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5882.338258] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5882.338399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5882.341748] LustreError: Skipped 1 previous similar message [ 5882.347143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5882.351522] Lustre: Skipped 1 previous similar message [ 5883.903725] LustreError: 241338:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5883.968796] Lustre: server umount lustre-OST0000 complete [ 5885.590668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 5890.279256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5890.454825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5892.848833] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5895.082464] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5902.695288] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.699558] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5902.699683] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5902.706314] Lustre: Skipped 1 previous similar message [ 5933.538479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5933.539115] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5933.541321] LustreError: Skipped 1 previous similar message [ 5933.541689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5933.545666] Lustre: Skipped 2 previous similar messages [ 5935.082835] LustreError: 242996:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5935.084806] LustreError: 242996:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5935.133650] Lustre: server umount lustre-OST0000 complete [ 5938.144927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5938.147144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5938.151129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5938.153280] Lustre: Skipped 1 previous similar message [ 5942.590706] LustreError: 243197:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5942.593496] LustreError: 243197:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5942.622239] Lustre: server umount lustre-MDT0000 complete [ 5943.902388] LustreError: 239710:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760566 with bad export cookie 14826533874699385075 [ 5943.904575] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5943.906189] LustreError: 239710:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5944.035949] Lustre: server umount lustre-MDT0001 complete [ 5947.780436] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 5948.808963] Key type lgssc unregistered [ 5948.940918] LNet: 243929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5948.944571] LNetError: 243929:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5948.953486] LNet: Removed LNI 192.168.206.151@tcp [ 5949.258847] Key type .llcrypt unregistered [ 5949.260546] Key type ._llcrypt unregistered [ 5958.636837] Key type ._llcrypt registered [ 5958.637983] Key type .llcrypt registered [ 5958.679362] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 5959.005671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5959.021536] alg: No test for adler32 (adler32-zlib) [ 5959.882464] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 5959.986813] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 5961.576195] Key type lgssc registered [ 5961.957816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5964.903574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5964.914974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5966.104681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5967.544901] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5970.753438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5970.901290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5972.452275] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5974.523231] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5975.917989] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5978.383864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5978.505890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5980.478437] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 5982.301557] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5992.871639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6008.289097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6008.295629] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6008.297910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6008.306844] Lustre: Skipped 1 previous similar message [ 6009.135160] LustreError: 248338:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6009.181372] Lustre: server umount lustre-OST0000 complete [ 6011.872743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6011.876220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6011.880408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6011.882539] Lustre: Skipped 1 previous similar message [ 6012.739859] LustreError: 248540:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6012.742157] LustreError: 248540:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6013.409082] LustreError: 245303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6013.410025] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6013.415283] LustreError: 245303:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6014.823149] LustreError: 248540:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6014.825333] LustreError: 248540:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6014.844543] Lustre: server umount lustre-MDT0000 complete [ 6016.142869] LustreError: 246020:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760639 with bad export cookie 11793515553692975280 [ 6016.145764] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6016.148438] LustreError: 246020:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6016.258328] Lustre: server umount lustre-MDT0001 complete [ 6020.090790] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 6021.311533] Key type lgssc unregistered [ 6021.459373] LNet: 249272:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6021.463048] LNetError: 249272:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6021.470564] LNet: Removed LNI 192.168.206.151@tcp [ 6021.797978] Key type .llcrypt unregistered [ 6021.799088] Key type ._llcrypt unregistered [ 6029.181091] Key type ._llcrypt registered [ 6029.182348] Key type .llcrypt registered [ 6029.224939] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6029.571973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6029.630076] alg: No test for adler32 (adler32-zlib) [ 6030.498832] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 6030.598684] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 6032.184149] Key type lgssc registered [ 6032.603865] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6035.373146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6035.381599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6036.593229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6038.008868] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6041.128980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6041.289901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6042.947244] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6044.976926] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6046.272071] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6048.903754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6049.075167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6051.520860] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6053.603837] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6061.359622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6071.780526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6071.780637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6071.788576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6073.520118] LustreError: 252856:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6073.556618] Lustre: server umount lustre-OST0000 complete [ 6076.899345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6076.899871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6076.903238] Lustre: Skipped 2 previous similar messages [ 6076.906833] Lustre: Skipped 2 previous similar messages [ 6082.018053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6082.020397] Lustre: Skipped 1 previous similar message [ 6087.137228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6087.139952] Lustre: Skipped 1 previous similar message [ 6089.184489] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6089.221173] LustreError: 253058:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6089.222701] LustreError: 253058:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6089.253802] Lustre: server umount lustre-MDT0000 complete [ 6090.522243] LustreError: 250430:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760713 with bad export cookie 1331693318691991759 [ 6090.525580] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6090.527253] LustreError: 250430:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6090.589309] LustreError: 253260:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6090.591072] LustreError: 253260:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6090.640088] Lustre: server umount lustre-MDT0001 complete [ 6094.165574] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 6095.312032] Key type lgssc unregistered [ 6095.438548] LNet: 253790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6095.441509] LNetError: 253790:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6095.448376] LNet: Removed LNI 192.168.206.151@tcp [ 6095.752319] Key type .llcrypt unregistered [ 6095.753459] Key type ._llcrypt unregistered [ 6104.879450] Key type ._llcrypt registered [ 6104.880594] Key type .llcrypt registered [ 6104.918168] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6105.268371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6105.297445] alg: No test for adler32 (adler32-zlib) [ 6106.152283] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 6106.236258] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 6107.816115] Key type lgssc registered [ 6108.172958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6110.891966] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6110.897857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6112.046587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6113.365196] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6116.462787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6116.585700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6117.908561] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6119.709397] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6120.895310] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6123.013762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6123.113529] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6124.934401] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6126.593852] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6136.423161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6147.043072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6147.044267] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6147.046046] LustreError: Skipped 1 previous similar message [ 6147.052065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6147.055047] Lustre: Skipped 1 previous similar message [ 6150.956179] LustreError: 258200:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6150.989596] Lustre: server umount lustre-OST0000 complete [ 6152.672762] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6152.676363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6152.681924] Lustre: Skipped 1 previous similar message [ 6152.684405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6157.280910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6157.281512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6158.330970] LustreError: 258402:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6158.333725] LustreError: 258402:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6158.360148] Lustre: server umount lustre-MDT0000 complete [ 6159.569611] LustreError: 255148:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760782 with bad export cookie 12997184597004515628 [ 6159.572847] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6159.574035] LustreError: 255148:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6159.641514] LustreError: 258603:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6159.643462] LustreError: 258603:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6159.687143] Lustre: server umount lustre-MDT0001 complete [ 6163.181349] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 6164.186229] Key type lgssc unregistered [ 6164.318439] LNet: 259134:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6164.320548] LNetError: 259134:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6164.327356] LNet: Removed LNI 192.168.206.151@tcp [ 6164.611857] Key type .llcrypt unregistered [ 6164.613229] Key type ._llcrypt unregistered [ 6171.674644] Key type ._llcrypt registered [ 6171.675881] Key type .llcrypt registered [ 6171.713538] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6172.048913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6172.096564] alg: No test for adler32 (adler32-zlib) [ 6172.962800] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 6173.059426] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 6174.640153] Key type lgssc registered [ 6174.995879] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6177.572938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6177.580935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6178.743937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6180.009697] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6182.681020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6182.799542] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6184.058450] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6185.722828] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6186.899170] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6188.993327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6189.026778] LustreError: 261943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6189.032569] LustreError: 261943:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6189.093889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6190.799751] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6192.168336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6192.485206] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6197.217274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6197.219516] LustreError: Skipped 1 previous similar message [ 6197.219543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6197.221184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6197.224133] Lustre: Skipped 1 previous similar message [ 6198.892035] LustreError: 262735:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6198.924658] Lustre: server umount lustre-OST0000 complete [ 6202.174305] LustreError: 262936:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6202.177239] LustreError: 262936:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6202.204659] Lustre: server umount lustre-MDT0000 complete [ 6202.337240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6202.338030] LustreError: 260319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6202.340871] Lustre: Skipped 1 previous similar message [ 6203.588800] LustreError: 260376:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760826 with bad export cookie 666904878797581341 [ 6203.591432] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6203.592294] LustreError: 260376:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6203.657488] LustreError: 263138:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6203.660102] LustreError: 263138:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6203.715747] Lustre: server umount lustre-MDT0001 complete [ 6207.486442] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 6208.637349] Key type lgssc unregistered [ 6208.769616] LNet: 263669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6208.773201] LNetError: 263669:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6208.782314] LNet: Removed LNI 192.168.206.151@tcp [ 6209.065253] Key type .llcrypt unregistered [ 6209.066678] Key type ._llcrypt unregistered [ 6212.786770] Key type ._llcrypt registered [ 6212.788350] Key type .llcrypt registered [ 6212.828737] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6218.612846] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 6224.725171] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6225.063585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6225.127053] alg: No test for adler32 (adler32-zlib) [ 6225.982326] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 6226.073535] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 6227.664143] Key type lgssc registered [ 6228.041619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6239.614990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6239.620814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6240.710545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6240.725260] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6240.759818] Lustre: lustre-MDT0000: new disk, initializing [ 6240.784194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6240.790603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6242.077363] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6246.202080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6246.239311] Lustre: 267140:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6246.262296] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6246.263851] Lustre: Skipped 1 previous similar message [ 6246.295765] Lustre: lustre-MDT0001: new disk, initializing [ 6246.326135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6246.337873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6246.341494] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6247.669109] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6250.818981] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6252.036062] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6254.376869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6254.485570] Lustre: lustre-OST0000: new disk, initializing [ 6254.488317] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6254.523351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6256.403190] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6259.512060] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6260.850957] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6261.249449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6261.252905] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6261.282503] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6261.942971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6263.275327] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6263.348703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6266.338182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6266.339387] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6266.341488] LustreError: Skipped 1 previous similar message [ 6266.342290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6266.354269] Lustre: Skipped 1 previous similar message [ 6270.065323] LustreError: 270077:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6270.110745] Lustre: server umount lustre-OST0000 complete [ 6271.457083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6271.457626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6271.461408] Lustre: Skipped 1 previous similar message [ 6271.465036] Lustre: Skipped 2 previous similar messages [ 6276.577025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6276.578780] Lustre: Skipped 1 previous similar message [ 6277.502173] LustreError: 270278:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6277.504469] LustreError: 270278:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6277.536533] Lustre: server umount lustre-MDT0000 complete [ 6278.755039] LustreError: 267132:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760901 with bad export cookie 10813329924510788769 [ 6278.756946] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6278.759137] LustreError: 267132:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6278.821566] LustreError: 270480:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6278.823129] LustreError: 270480:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6278.872042] Lustre: server umount lustre-MDT0001 complete [ 6281.565974] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:15:04 (1769760904) [ 6283.702685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6283.886280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6285.286526] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6288.115493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6288.247890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6289.597486] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6291.449895] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6292.782256] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6295.162774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6297.259385] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6299.140858] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6313.953478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6313.954891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6313.956073] LustreError: Skipped 1 previous similar message [ 6313.957936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6313.959860] Lustre: Skipped 1 previous similar message [ 6318.767077] LustreError: 273410:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6318.769261] LustreError: 273410:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6318.807352] Lustre: server umount lustre-OST0000 complete [ 6323.169914] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6323.172558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6323.176127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6323.177669] Lustre: Skipped 1 previous similar message [ 6326.396599] LustreError: 273611:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6326.398539] LustreError: 273611:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6326.424987] Lustre: server umount lustre-MDT0000 complete [ 6327.832224] LustreError: 272762:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769760950 with bad export cookie 10813329924510789637 [ 6327.834970] LustreError: 271869:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 6327.835913] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6327.836239] LustreError: 272762:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6327.958962] Lustre: server umount lustre-MDT0001 complete [ 6331.479789] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 6332.478332] Key type lgssc unregistered [ 6332.598481] LNet: 274345:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6332.601379] LNetError: 274345:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6332.609505] LNet: Removed LNI 192.168.206.151@tcp [ 6332.904055] Key type .llcrypt unregistered [ 6332.905509] Key type ._llcrypt unregistered [ 6340.935042] Key type ._llcrypt registered [ 6340.936325] Key type .llcrypt registered [ 6340.972864] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 6346.814814] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6347.116822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6347.127784] alg: No test for adler32 (adler32-zlib) [ 6347.966325] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 6348.046486] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 6349.632554] Key type lgssc registered [ 6350.024129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6365.105536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6365.111767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6366.195783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6366.207489] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6366.242800] Lustre: lustre-MDT0000: new disk, initializing [ 6366.263641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6366.270223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6367.569256] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6371.601488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6371.640312] Lustre: 278105:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6371.661756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6371.663315] Lustre: Skipped 1 previous similar message [ 6371.696968] Lustre: lustre-MDT0001: new disk, initializing [ 6371.727845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6371.739787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6371.743202] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6373.003156] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6376.225422] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6377.450415] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6379.747386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6379.869406] Lustre: lustre-OST2710: new disk, initializing [ 6379.871729] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 6379.906311] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 6382.025326] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6384.656873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 6384.661325] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 6384.693515] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 6385.372870] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 6387.888535] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6387.957239] Lustre: lustre-OST03e8: new disk, initializing [ 6387.964739] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 6388.001903] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 6390.017813] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6391.862795] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 6394.371659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 6394.376703] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 6394.407947] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 6394.581133] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 6394.677219] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6396.080857] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 6396.160929] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6397.560802] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 6397.638868] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6399.136352] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 6399.210854] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6402.528910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6402.532552] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6402.536841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6404.580304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6404.581185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6404.585904] Lustre: Skipped 1 previous similar message [ 6404.593186] Lustre: Skipped 3 previous similar messages [ 6406.468194] LustreError: 282432:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6406.503195] Lustre: server umount lustre-MDT0000 complete [ 6407.759133] LustreError: 280793:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761030 with bad export cookie 8919928801709074798 [ 6407.762486] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6407.763107] LustreError: 280793:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6407.825459] LustreError: 282633:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6407.828335] LustreError: 282633:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6407.876146] Lustre: server umount lustre-MDT0001 complete [ 6419.896094] LustreError: 282833:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6419.903964] LustreError: 282833:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6419.940566] Lustre: server umount lustre-OST2710 complete [ 6431.541829] LustreError: 283036:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6431.544385] LustreError: 283036:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6431.593354] Lustre: server umount lustre-OST03e8 complete [ 6438.175638] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 6441.417930] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6454.208451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6454.300408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6454.312900] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6454.351952] Lustre: lustre-MDT0000: new disk, initializing [ 6454.381737] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6454.389131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6455.729564] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6460.085259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6460.126424] Lustre: 285818:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6460.164268] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6460.246935] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6461.588891] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6464.749141] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6465.987818] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6468.487780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6468.586287] Lustre: lustre-OST0000: new disk, initializing [ 6468.588386] Lustre: Skipped 1 previous similar message [ 6468.590641] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6468.592819] Lustre: Skipped 2 previous similar messages [ 6468.617981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6468.619865] Lustre: Skipped 1 previous similar message [ 6470.543218] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6473.537624] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6473.719965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6473.722342] Lustre: Skipped 1 previous similar message [ 6473.723987] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6473.757630] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6474.839722] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6474.914390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6476.160988] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6476.229085] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6478.817644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6478.819222] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6478.822427] LustreError: Skipped 1 previous similar message [ 6478.826269] Lustre: Skipped 1 previous similar message [ 6478.827102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6478.831021] Lustre: Skipped 1 previous similar message [ 6482.933085] LustreError: 288747:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6482.937788] LustreError: 288747:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6482.977670] Lustre: server umount lustre-OST0000 complete [ 6485.984809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6485.987060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6485.990634] Lustre: Skipped 1 previous similar message [ 6485.992518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6490.326962] Lustre: server umount lustre-MDT0000 complete [ 6491.592084] LustreError: 285810:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761114 with bad export cookie 8919928801709076457 [ 6491.594648] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6491.595864] LustreError: 285810:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6491.654262] LustreError: 289149:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6491.656205] LustreError: 289149:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 6491.719387] Lustre: server umount lustre-MDT0001 complete [ 6494.445592] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:18:37 (1769761117) [ 6494.906041] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 6495.472492] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:18:38 (1769761118) [ 6496.489839] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6505.730213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6505.846231] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6505.857580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6505.922526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6507.253039] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6510.265583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6510.302252] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6510.311668] Lustre: 291242:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6511.746712] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6513.514460] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6514.655960] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6527.914582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6528.002087] Lustre: Denying initial registration attempt from nid 192.168.206.151@tcp, specified as failover [ 6528.005724] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 6528.008291] LustreError: 292853:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -99 [ 6528.011450] LustreError: 292853:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 6528.014109] LustreError: 292853:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 6528.042942] Lustre: server umount lustre-OST0000 complete [ 6528.044397] LustreError: 292853:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 6531.041071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6531.048512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6531.055120] Lustre: Skipped 1 previous similar message [ 6531.057775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6531.060186] Lustre: Skipped 2 previous similar messages [ 6545.376088] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6545.410087] LustreError: 293220:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 6545.412383] LustreError: 293220:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6545.435390] Lustre: server umount lustre-MDT0000 complete [ 6546.702306] LustreError: 291234:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761169 with bad export cookie 8919928801709077339 [ 6546.705018] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6546.708924] LustreError: 291234:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6546.820516] Lustre: server umount lustre-MDT0001 complete [ 6550.426292] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 6551.426076] Key type lgssc unregistered [ 6551.556310] LNet: 293954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6551.558764] LNetError: 293954:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6551.569412] LNet: Removed LNI 192.168.206.151@tcp [ 6551.905083] Key type .llcrypt unregistered [ 6551.906439] Key type ._llcrypt unregistered [ 6555.543798] Key type ._llcrypt registered [ 6555.544839] Key type .llcrypt registered [ 6555.585612] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6556.137757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6556.146838] alg: No test for adler32 (adler32-zlib) [ 6557.031938] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 6557.130774] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 6558.712146] Key type lgssc registered [ 6559.140211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6594.330507] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6597.214828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6597.221755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6598.306195] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6598.312919] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6598.369686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6599.699886] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6602.731393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6602.781717] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6602.796502] Lustre: 296662:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6602.900479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6604.301600] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6606.086400] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6607.269952] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6620.770084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6620.841156] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6620.908539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6623.157523] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6625.462218] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6626.267171] LustreError: 298929:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6626.320408] Lustre: server umount lustre-OST0000 complete [ 6628.832781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6628.834973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6628.839073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6630.897786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6630.898710] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6636.002097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6636.004884] Lustre: Skipped 2 previous similar messages [ 6640.097264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6640.099447] Lustre: Skipped 1 previous similar message [ 6642.144193] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6642.184546] LustreError: 299130:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 6642.186327] LustreError: 299130:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6642.212918] Lustre: server umount lustre-MDT0000 complete [ 6643.654349] LustreError: 296653:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761266 with bad export cookie 18344236416127501205 [ 6643.658029] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6643.658558] LustreError: 296653:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6643.718520] LustreError: 299336:0:(obd_class.h:479:obd_check_dev()) Device 7 not setup [ 6643.720956] LustreError: 299336:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6643.796611] Lustre: server umount lustre-MDT0001 complete [ 6647.652294] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 6648.709197] Key type lgssc unregistered [ 6648.841368] LNet: 299867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6648.844129] LNetError: 299867:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6648.851404] LNet: Removed LNI 192.168.206.151@tcp [ 6649.171100] Key type .llcrypt unregistered [ 6649.172695] Key type ._llcrypt unregistered [ 6658.853509] Key type ._llcrypt registered [ 6658.854865] Key type .llcrypt registered [ 6658.891311] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6659.220152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6659.257047] alg: No test for adler32 (adler32-zlib) [ 6660.119413] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 6660.202210] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 6661.792147] Key type lgssc registered [ 6662.205799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6665.211286] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6665.217961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6666.379397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6667.796563] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6670.947369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6671.100815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6672.516254] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6674.520445] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6675.974895] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6678.316919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6678.434977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6680.249977] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6682.060266] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6696.929935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6696.930849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6696.931643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6696.933145] Lustre: Skipped 1 previous similar message [ 6697.835077] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 6697.837240] Lustre: Skipped 2 previous similar messages [ 6700.668833] LustreError: 303507:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6700.692796] Lustre: server umount lustre-MDT0000 complete [ 6701.940186] LustreError: 301221:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761324 with bad export cookie 6001523400125308075 [ 6701.943865] LustreError: 301221:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6701.944232] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6702.048896] LustreError: 301236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6702.049164] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6702.053716] LustreError: 301236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6702.057816] Lustre: Skipped 1 previous similar message [ 6702.062364] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6702.954356] LustreError: 301236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6706.080737] LustreError: 301236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6706.080948] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6706.088635] Lustre: Skipped 1 previous similar message [ 6711.201057] LustreError: 301236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6711.201304] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6711.205163] LustreError: 301236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6711.210096] Lustre: Skipped 1 previous similar message [ 6716.384104] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6716.384903] LustreError: 301236:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6716.392358] LustreError: 301236:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6716.420138] LustreError: 303709:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6716.422286] LustreError: 303709:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6716.475716] Lustre: server umount lustre-MDT0001 complete [ 6718.751714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 6721.703529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6726.624642] LustreError: 300550:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b1e330f5 x1855729207106816/t0(0) o250->MGC192.168.206.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6726.730978] LustreError: 302846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6726.759628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6726.780953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 6728.095371] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6731.099135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6731.226449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6732.566763] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6734.479915] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6745.528790] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 10 sec [ 6746.599049] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6746.818625] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6747.370836] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6751.716037] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6751.716973] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6751.724549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6751.727014] Lustre: Skipped 3 previous similar messages [ 6754.672417] LustreError: 306035:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6754.676086] LustreError: 306035:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6754.718397] Lustre: server umount lustre-OST0000 complete [ 6756.834929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6756.836198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6756.840574] Lustre: Skipped 2 previous similar messages [ 6762.048284] LustreError: 306237:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6762.051092] LustreError: 306237:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6762.079190] Lustre: server umount lustre-MDT0000 complete [ 6763.425893] LustreError: 301223:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761386 with bad export cookie 6001523400125315194 [ 6763.428825] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6763.429641] LustreError: 301223:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6763.537725] Lustre: server umount lustre-MDT0001 complete [ 6766.904454] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 6767.912361] Key type lgssc unregistered [ 6768.037145] LNet: 306970:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6768.040412] LNetError: 306970:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6768.051443] LNet: Removed LNI 192.168.206.151@tcp [ 6768.349688] Key type .llcrypt unregistered [ 6768.350587] Key type ._llcrypt unregistered [ 6777.313078] Key type ._llcrypt registered [ 6777.314054] Key type .llcrypt registered [ 6777.352271] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6777.650533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6777.658156] alg: No test for adler32 (adler32-zlib) [ 6778.533042] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 6778.622466] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 6780.200147] Key type lgssc registered [ 6780.586078] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6783.141253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6783.149351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6784.307505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6785.622965] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6788.562032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6788.678437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6789.980710] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6791.718465] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6792.963302] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6798.049809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6798.055418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6798.057558] Lustre: Skipped 1 previous similar message [ 6800.059243] LustreError: 309733:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6800.085072] Lustre: server umount lustre-MDT0000 complete [ 6801.261938] LustreError: 308323:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761424 with bad export cookie 11729339217763868468 [ 6801.265551] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6801.266807] LustreError: 308323:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6801.328683] LustreError: 309935:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6801.330749] LustreError: 309935:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6801.381598] Lustre: server umount lustre-MDT0001 complete [ 6806.012195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6806.101106] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6806.112579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6806.182328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6807.422266] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6810.197969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6810.231382] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6810.242360] Lustre: 310704:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6810.329954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6811.663332] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6813.678132] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6814.900967] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6817.605126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6817.684478] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 6817.688163] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 6817.692302] LustreError: 312359:0:(tgt_mount.c:2579:server_fill_super()) Unable to start targets: -2 [ 6817.695765] LustreError: 312359:0:(tgt_mount.c:2066:server_put_super()) no obd lustre-OST0000 [ 6817.698783] LustreError: 312359:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 6817.724164] Lustre: server umount lustre-OST0000 complete [ 6817.725787] LustreError: 312359:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 6820.732225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6820.793935] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6820.857499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6821.541327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 6822.924199] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6824.825451] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6827.731357] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6827.761341] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6827.791057] Lustre: lustre-OST0001: new disk, initializing [ 6827.792959] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6829.769939] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6831.504833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6831.509049] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6831.545755] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6831.681498] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6836.705627] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6836.705718] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6836.707963] LustreError: Skipped 1 previous similar message [ 6836.710911] Lustre: Skipped 1 previous similar message [ 6836.712724] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6836.715149] Lustre: Skipped 1 previous similar message [ 6838.768092] LustreError: 314471:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6838.770776] LustreError: 314471:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6838.801738] Lustre: server umount lustre-OST0001 complete [ 6841.826876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6841.827841] LustreError: 312826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6841.828784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6841.831438] Lustre: Skipped 1 previous similar message [ 6841.840145] LustreError: 312826:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6846.128096] LustreError: 314672:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6846.130395] LustreError: 314672:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6846.166857] Lustre: server umount lustre-OST0000 complete [ 6851.552825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6851.556092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6851.560487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6851.562099] Lustre: Skipped 1 previous similar message [ 6853.568054] LustreError: 314873:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 6853.570074] LustreError: 314873:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6853.598704] Lustre: server umount lustre-MDT0000 complete [ 6854.808112] LustreError: 310695:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761477 with bad export cookie 11729339217763869007 [ 6854.811851] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6854.813093] LustreError: 310695:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6854.937324] Lustre: server umount lustre-MDT0001 complete [ 6858.180987] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 6859.158305] Key type lgssc unregistered [ 6859.281358] LNet: 315606:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6859.283932] LNetError: 315606:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6859.296414] LNet: Removed LNI 192.168.206.151@tcp [ 6859.581705] Key type .llcrypt unregistered [ 6859.582991] Key type ._llcrypt unregistered [ 6888.687242] Key type ._llcrypt registered [ 6888.688088] Key type .llcrypt registered [ 6888.719070] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 6894.611968] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 6895.089789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6895.103543] alg: No test for adler32 (adler32-zlib) [ 6895.957258] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 6896.040777] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 6897.624143] Key type lgssc registered [ 6897.989643] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6909.014506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6909.020699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6910.101659] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6910.112553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6910.149829] Lustre: lustre-MDT0000: new disk, initializing [ 6910.177750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6910.188194] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6911.478540] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6915.636973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6915.675904] Lustre: 320470:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6915.701878] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6915.703609] Lustre: Skipped 1 previous similar message [ 6915.735274] Lustre: lustre-MDT0001: new disk, initializing [ 6915.763369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6915.776796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6915.780681] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6917.065990] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6920.181162] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6921.308903] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6923.701225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6923.799068] Lustre: lustre-OST0000: new disk, initializing [ 6923.802296] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6923.839520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6925.764790] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6926.463371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6926.467899] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6926.505335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6928.949757] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6930.358018] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6930.434934] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6931.777129] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6931.849432] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6937.058085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6937.058710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6937.060755] LustreError: Skipped 1 previous similar message [ 6937.064867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6937.068102] Lustre: Skipped 1 previous similar message [ 6938.542086] LustreError: 323400:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6938.578604] Lustre: server umount lustre-OST0000 complete [ 6941.664949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6941.666985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6941.671070] Lustre: Skipped 1 previous similar message [ 6941.673102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6945.915990] LustreError: 323601:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6945.918323] LustreError: 323601:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6945.945865] Lustre: server umount lustre-MDT0000 complete [ 6947.211330] LustreError: 320462:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761570 with bad export cookie 816207363284615917 [ 6947.214773] LustreError: 320462:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6947.215039] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6947.280924] LustreError: 323802:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6947.283440] LustreError: 323802:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6947.345911] Lustre: server umount lustre-MDT0001 complete [ 6950.103394] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 03:26:13 (1769761573) [ 6952.337952] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 03:26:15 (1769761575) [ 6955.011843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6955.177123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6956.367729] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6959.018933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6960.363642] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6962.052389] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6963.213153] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6965.405915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6965.512500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6965.515141] Lustre: Skipped 1 previous similar message [ 6967.251078] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 6969.016078] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6987.852591] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 6988.416813] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6988.947394] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 6989.521387] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 6990.213765] Lustre: Failing over lustre-MDT0000 [ 6990.395246] LustreError: 327270:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6990.397613] LustreError: 327270:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6990.431254] Lustre: server umount lustre-MDT0000 complete [ 6991.329951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6991.330920] LustreError: 324808:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6991.337441] Lustre: Skipped 3 previous similar messages [ 6991.342173] LustreError: 324808:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6994.794648] LustreError: 326978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6994.799530] LustreError: 326978:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6996.449350] LustreError: 325525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6996.454544] LustreError: 325525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6999.908981] LustreError: 324808:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7003.371320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7003.415174] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7003.499995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7003.531805] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7004.693223] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7005.030408] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7008.741846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7008.752326] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7008.767994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7009.855750] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7010.384927] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7012.136152] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7012.748740] Lustre: Failing over lustre-MDT0000 [ 7012.860020] LustreError: 328352:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7012.861614] LustreError: 328352:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7012.886553] Lustre: server umount lustre-MDT0000 complete [ 7013.858713] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7013.859288] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7013.860452] LustreError: 324814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7013.860458] LustreError: 324814:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7013.871381] Lustre: Skipped 2 previous similar messages [ 7024.097104] LustreError: 324813:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7024.104690] LustreError: 324813:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 7025.589230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7025.633446] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7025.715752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7025.734527] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7026.997231] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7030.629869] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7030.757929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7030.760199] Lustre: Skipped 2 previous similar messages [ 7030.767407] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 7030.783465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7031.847572] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7032.369159] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7035.873963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7035.874568] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7035.877653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7035.881552] Lustre: Skipped 4 previous similar messages [ 7040.703853] LustreError: 329384:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7040.706334] LustreError: 329384:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7040.732432] Lustre: server umount lustre-MDT0000 complete [ 7040.993979] LustreError: 324814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7041.000445] LustreError: 324814:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7041.964043] LustreError: 327272:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761664 with bad export cookie 816207363284618598 [ 7041.966640] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7041.968633] LustreError: 327272:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7042.073546] Lustre: server umount lustre-MDT0001 complete [ 7053.585325] Lustre: server umount lustre-OST0000 complete [ 7058.513629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7058.669132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7059.852959] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7062.425217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7063.697307] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7065.252249] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7066.291992] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7068.424151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7069.544793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7070.495000] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7072.281099] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7079.904816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7079.907493] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7079.911461] Lustre: Skipped 2 previous similar messages [ 7079.913409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7084.654654] LustreError: 332855:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7084.657010] LustreError: 332855:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 7084.689437] Lustre: server umount lustre-OST0000 complete [ 7088.098268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7088.100612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7088.101556] LustreError: Skipped 1 previous similar message [ 7088.104031] Lustre: Skipped 3 previous similar messages [ 7098.337549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7098.339945] Lustre: Skipped 3 previous similar messages [ 7100.384097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7100.449611] Lustre: server umount lustre-MDT0000 complete [ 7101.713301] LustreError: 330596:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761724 with bad export cookie 816207363284619319 [ 7101.715228] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7101.717419] LustreError: 330596:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7101.818426] Lustre: server umount lustre-MDT0001 complete [ 7105.104983] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 7106.057365] Key type lgssc unregistered [ 7106.191328] LNet: 333791:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7106.193369] LNetError: 333791:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7106.206303] LNet: Removed LNI 192.168.206.151@tcp [ 7106.505374] Key type .llcrypt unregistered [ 7106.506306] Key type ._llcrypt unregistered [ 7115.487609] Key type ._llcrypt registered [ 7115.488566] Key type .llcrypt registered [ 7115.522961] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7115.829527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7115.879493] alg: No test for adler32 (adler32-zlib) [ 7116.719852] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 7116.793267] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 7118.368142] Key type lgssc registered [ 7118.765168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7121.210853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7121.216524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7122.374727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7123.682264] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7126.500925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7126.611740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7127.819620] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7129.595034] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7130.716423] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7132.758757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7132.860787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7134.698368] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7136.323790] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7147.175630] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7157.216784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7157.219192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7157.223100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7157.731932] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7157.735574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7157.737227] Lustre: Skipped 1 previous similar message [ 7161.851381] LustreError: 337526:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7161.876303] Lustre: server umount lustre-MDT0000 complete [ 7162.849205] LustreError: 335265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7162.854687] LustreError: 335265:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7163.035964] LustreError: 335245:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761785 with bad export cookie 572294364864873315 [ 7163.038660] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7163.039574] LustreError: 335245:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7163.094112] LustreError: 337729:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7163.096958] LustreError: 337729:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7163.148277] Lustre: server umount lustre-MDT0001 complete [ 7174.579348] LustreError: 337930:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7174.581321] LustreError: 337930:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7174.616951] Lustre: server umount lustre-OST0000 complete [ 7179.147056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7179.284193] LustreError: 338658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7179.308836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7180.445148] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7183.041258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7184.510472] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7186.413466] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7187.619334] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7189.874613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7189.997873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7190.000295] Lustre: Skipped 1 previous similar message [ 7191.910913] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7193.736833] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7204.326411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7219.168791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7219.171123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7219.174539] Lustre: Skipped 1 previous similar message [ 7219.176849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7222.140417] LustreError: 341022:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7222.142561] LustreError: 341022:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7222.169593] Lustre: server umount lustre-MDT0000 complete [ 7223.324085] LustreError: 338638:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761846 with bad export cookie 572294364864874414 [ 7223.327606] LustreError: 338638:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7223.327698] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7223.442331] Lustre: server umount lustre-MDT0001 complete [ 7234.865328] LustreError: 341426:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7234.867220] LustreError: 341426:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 7234.902338] Lustre: server umount lustre-OST0000 complete [ 7239.545589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7239.671889] LustreError: 342253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7239.692525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7240.821554] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7243.328581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7244.589861] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7246.257597] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7247.332478] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7249.459170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7250.603803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 7251.351293] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7253.015886] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7256.032756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7256.034699] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7256.038178] Lustre: Skipped 2 previous similar messages [ 7256.040171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7256.041997] Lustre: Skipped 2 previous similar messages [ 7259.109232] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7260.076566] LustreError: 344486:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7260.079112] LustreError: 344486:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7260.119394] Lustre: server umount lustre-OST0000 complete [ 7264.225699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7264.226315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7264.231788] Lustre: Skipped 1 previous similar message [ 7264.233884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7264.235512] Lustre: Skipped 3 previous similar messages [ 7267.474701] Lustre: server umount lustre-MDT0000 complete [ 7268.569997] LustreError: 342232:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761891 with bad export cookie 572294364864875422 [ 7268.572508] LustreError: 342232:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7268.573447] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7268.673652] Lustre: server umount lustre-MDT0001 complete [ 7272.025152] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 7273.017780] Key type lgssc unregistered [ 7273.156343] LNet: 345426:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7273.158546] LNetError: 345426:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7273.164320] LNet: Removed LNI 192.168.206.151@tcp [ 7273.490043] Key type .llcrypt unregistered [ 7273.491040] Key type ._llcrypt unregistered [ 7282.839308] Key type ._llcrypt registered [ 7282.840585] Key type .llcrypt registered [ 7282.877662] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7283.245555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7283.252825] alg: No test for adler32 (adler32-zlib) [ 7284.096482] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 7284.168633] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 7285.744171] Key type lgssc registered [ 7286.064674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7288.378355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7288.383963] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7288.386443] LustreError: 346822:0:(osd_handler.c:8575:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 7288.393857] LustreError: 346822:0:(obd_config.c:859:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 7288.395890] LustreError: 346822:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 7288.397923] LustreError: 346822:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 7288.400260] LustreError: 346822:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7290.707090] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7290.709365] LustreError: 347219:0:(osd_handler.c:8575:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 7290.716796] LustreError: 347219:0:(obd_config.c:859:class_setup()) setup lustre-OST0000-osd failed (-22) [ 7290.719260] LustreError: 347219:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 7290.721131] LustreError: 347219:0:(tgt_mount.c:2547:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 7290.722981] LustreError: 347219:0:(super25.c:190:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7296.187018] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 7297.085077] Key type lgssc unregistered [ 7297.200335] LNet: 348201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7297.201898] LNetError: 348201:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7297.211244] LNet: Removed LNI 192.168.206.151@tcp [ 7297.460410] Key type .llcrypt unregistered [ 7297.461694] Key type ._llcrypt unregistered [ 7303.525840] Key type ._llcrypt registered [ 7303.527076] Key type .llcrypt registered [ 7303.569904] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 7309.423964] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7309.859084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7309.878392] alg: No test for adler32 (adler32-zlib) [ 7310.719367] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 7310.791931] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 7312.368186] Key type lgssc registered [ 7312.695696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7323.268482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7323.273052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7324.351240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7324.363191] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7324.395174] Lustre: lustre-MDT0000: new disk, initializing [ 7324.416982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7324.424507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7325.638997] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7329.494347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7329.530939] Lustre: 351293:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7329.553321] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7329.554861] Lustre: Skipped 1 previous similar message [ 7329.586500] Lustre: lustre-MDT0001: new disk, initializing [ 7329.617105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7329.629985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7329.632790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7330.780356] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7333.592492] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7334.671102] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7336.670675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7336.760313] Lustre: lustre-OST0000: new disk, initializing [ 7336.761854] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7336.786691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7338.389778] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7341.229918] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7342.074978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7342.079237] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7342.118308] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7342.600653] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7342.664939] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7343.928600] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7344.002974] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7347.169786] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7347.171408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7347.171787] LustreError: Skipped 1 previous similar message [ 7347.176103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7347.178436] Lustre: Skipped 1 previous similar message [ 7350.701141] LustreError: 354223:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7350.736341] Lustre: server umount lustre-OST0000 complete [ 7352.289616] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7352.290278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7352.293155] Lustre: Skipped 2 previous similar messages [ 7357.408960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7357.410457] Lustre: Skipped 2 previous similar messages [ 7362.529179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7362.530778] Lustre: Skipped 1 previous similar message [ 7366.112086] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7366.147969] LustreError: 354425:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7366.149990] LustreError: 354425:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7366.172717] Lustre: server umount lustre-MDT0000 complete [ 7367.310069] LustreError: 353033:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769761990 with bad export cookie 5075858690474335402 [ 7367.313025] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7367.314720] LustreError: 353033:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7367.380996] LustreError: 354628:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7367.383420] LustreError: 354628:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7367.429736] Lustre: server umount lustre-MDT0001 complete [ 7369.942105] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 7370.437366] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 03:33:13 (1769761993) [ 7372.450268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7372.622680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7373.750175] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7376.259880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7377.593633] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7379.219522] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7380.414787] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7382.559221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7382.661780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7382.664228] Lustre: Skipped 1 previous similar message [ 7384.503787] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7386.101301] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7388.157304] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7388.216259] Lustre: lustre-OST0001: new disk, initializing [ 7388.218335] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7388.436713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7388.440246] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7388.478115] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7389.978304] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7392.922568] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7397.798981] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7397.802495] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7397.803758] Lustre: Skipped 1 previous similar message [ 7400.750534] LustreError: 358541:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 7400.752530] LustreError: 358541:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7400.770271] Lustre: server umount lustre-OST0001 complete [ 7402.980058] LustreError: 357569:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7402.985415] LustreError: 357569:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7408.103835] LustreError: 356940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7408.109129] LustreError: 356940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7412.704703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7412.706477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7412.709704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7412.712015] Lustre: Skipped 2 previous similar messages [ 7413.220224] LustreError: 356940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7413.225915] LustreError: 356940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7418.342074] LustreError: 356937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7418.346546] LustreError: 356937:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7418.733090] LustreError: 358764:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7418.735369] LustreError: 358764:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7418.768348] Lustre: server umount lustre-OST0000 complete [ 7422.432695] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7422.434501] LustreError: Skipped 1 previous similar message [ 7422.435556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7422.438875] Lustre: Skipped 1 previous similar message [ 7426.134153] Lustre: server umount lustre-MDT0000 complete [ 7427.394441] LustreError: 355328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762050 with bad export cookie 5075858690474336270 [ 7427.398529] LustreError: 355328:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7427.398645] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7433.535176] LustreError: 359167:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 7433.537163] LustreError: 359167:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 7433.586807] Lustre: server umount lustre-MDT0001 complete [ 7437.020339] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 7438.178193] Key type lgssc unregistered [ 7438.299418] LNet: 359699:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7438.301927] LNetError: 359699:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7438.308333] LNet: Removed LNI 192.168.206.151@tcp [ 7438.625253] Key type .llcrypt unregistered [ 7438.626267] Key type ._llcrypt unregistered [ 7456.529277] Key type ._llcrypt registered [ 7456.530285] Key type .llcrypt registered [ 7456.574081] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7457.107865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7457.119787] alg: No test for adler32 (adler32-zlib) [ 7457.976205] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 7458.069229] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 7459.648123] Key type lgssc registered [ 7460.047410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7462.773177] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7462.778993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7463.868369] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7463.875715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7463.935371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7465.303697] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7468.171811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7468.205058] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7468.215873] Lustre: 362451:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7468.316730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7469.618833] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7471.350100] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7472.493443] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7473.632844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7473.635111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7473.640319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7476.705142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7476.705566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7479.304987] LustreError: 363816:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 7479.327807] Lustre: server umount lustre-MDT0000 complete [ 7480.587449] LustreError: 362444:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762103 with bad export cookie 9522634886441816655 [ 7480.589711] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7480.591807] LustreError: 362444:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7480.663615] LustreError: 364018:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 7480.665431] LustreError: 364018:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7480.716974] Lustre: server umount lustre-MDT0001 complete [ 7483.718528] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 03:35:06 (1769762106) [ 7485.878034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7486.045429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7487.405165] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7490.172317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7490.295960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7491.535417] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7493.225161] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7494.417967] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7496.644236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7496.707023] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7496.769681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7498.606179] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7500.425379] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7504.114285] Lustre: Permanently deactivating lustre-OST0000 [ 7504.119977] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7506.350549] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7507.308379] LustreError: 367278:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 7507.312055] LustreError: 367278:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 7507.314235] LustreError: 367278:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7507.901611] LustreError: 367326:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 7507.904516] LustreError: 367326:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7514.606782] LustreError: 367424:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7514.608967] LustreError: 367424:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7514.651308] Lustre: server umount lustre-OST0000 complete [ 7515.562306] LustreError: 367577:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 7515.566499] LustreError: 367577:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 7520.738154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7520.739549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7520.742064] Lustre: Skipped 1 previous similar message [ 7520.746573] Lustre: Skipped 2 previous similar messages [ 7525.857511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7525.859470] Lustre: Skipped 1 previous similar message [ 7526.884092] LustreError: 367674:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7526.885776] LustreError: 367674:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7526.909373] Lustre: server umount lustre-MDT0000 complete [ 7528.160388] LustreError: 367080:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762151 with bad export cookie 9522634886441817222 [ 7528.163785] LustreError: 367080:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7528.164098] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7538.660261] LustreError: 367875:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7538.663199] LustreError: 367875:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7538.711650] Lustre: server umount lustre-MDT0001 complete [ 7541.580167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7542.931617] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7543.892605] LustreError: 368716:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 7546.024800] Lustre: 369054:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.151@tcp [ 7546.670419] LustreError: 369151:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7546.673682] LustreError: 369151:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7546.699213] Lustre: server umount MGS complete [ 7550.367956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7550.517271] LustreError: 369777:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7550.522886] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7550.523342] LustreError: 369777:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7550.526272] Lustre: Skipped 1 previous similar message [ 7550.549837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7551.854780] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7554.670382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7554.764880] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7556.110972] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7557.843055] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7558.987076] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7559.603513] Lustre: Permanently reactivating lustre-OST0000 [ 7559.608775] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7559.611111] Lustre: Skipped 2 previous similar messages [ 7580.128091] LustreError: 371129:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 7582.625747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7582.732381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7582.735560] Lustre: Skipped 1 previous similar message [ 7584.595587] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7586.482968] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7600.096856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7600.099446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7600.103836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7601.125122] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7604.206192] LustreError: 372960:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7604.208111] LustreError: 372960:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 7604.239424] Lustre: server umount lustre-OST0000 complete [ 7606.241992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7606.242315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7606.246486] Lustre: Skipped 1 previous similar message [ 7611.361172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7611.363861] Lustre: Skipped 5 previous similar messages [ 7611.541913] Lustre: server umount lustre-MDT0000 complete [ 7612.801490] LustreError: 371125:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762235 with bad export cookie 9522634886441818258 [ 7612.805227] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7612.807087] LustreError: 371125:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7612.938661] Lustre: server umount lustre-MDT0001 complete [ 7616.346775] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 7617.367364] Key type lgssc unregistered [ 7617.493348] LNet: 373900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7617.495771] LNetError: 373900:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7617.503268] LNet: Removed LNI 192.168.206.151@tcp [ 7617.808882] Key type .llcrypt unregistered [ 7617.809923] Key type ._llcrypt unregistered [ 7624.032824] Key type ._llcrypt registered [ 7624.033711] Key type .llcrypt registered [ 7624.070044] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 7630.179689] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7630.512462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7630.565877] alg: No test for adler32 (adler32-zlib) [ 7631.414661] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 7631.502896] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 7633.088122] Key type lgssc registered [ 7633.443844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7644.816767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7644.824165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7645.920048] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7645.935324] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7645.973857] Lustre: lustre-MDT0000: new disk, initializing [ 7646.000695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7646.007960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7647.264236] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7651.444083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7651.490520] Lustre: 376992:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7651.521122] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7651.522835] Lustre: Skipped 1 previous similar message [ 7651.557924] Lustre: lustre-MDT0001: new disk, initializing [ 7651.593289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7651.606620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7651.610547] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7652.927169] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7655.932602] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7657.210965] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7659.507160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7659.600978] Lustre: lustre-OST0000: new disk, initializing [ 7659.602865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7659.630732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7661.446494] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7663.614441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7663.617503] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7663.665991] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7664.531062] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7665.834240] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7665.910772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7667.233858] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7667.310402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7668.707831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7668.707905] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7668.709293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7668.711632] LustreError: Skipped 1 previous similar message [ 7668.717177] Lustre: Skipped 1 previous similar message [ 7669.998160] LustreError: 379923:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7670.032850] Lustre: server umount lustre-OST0000 complete [ 7672.288759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7672.291367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7672.296152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7672.297947] Lustre: Skipped 1 previous similar message [ 7673.825336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7673.826130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7677.438332] LustreError: 380124:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7677.440340] LustreError: 380124:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7677.470552] Lustre: server umount lustre-MDT0000 complete [ 7678.744301] LustreError: 376985:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762301 with bad export cookie 5992887278237105354 [ 7678.747957] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7678.749194] LustreError: 376985:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7678.823814] LustreError: 380327:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7678.826500] LustreError: 380327:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7678.887396] Lustre: server umount lustre-MDT0001 complete [ 7681.692726] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 7682.226358] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 03:38:25 (1769762305) [ 7684.516058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7684.705075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7686.018746] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7688.880886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7688.996176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7690.316222] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7692.020830] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7693.210286] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7695.464952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7697.419763] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7699.166969] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7699.813036] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 7710.634420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 7714.272805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7714.275822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7714.281272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7714.284246] Lustre: Skipped 1 previous similar message [ 7718.127120] LustreError: 383500:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7718.129019] LustreError: 383500:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7718.169313] Lustre: server umount lustre-OST0000 complete [ 7719.905090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7719.907884] LustreError: Skipped 1 previous similar message [ 7719.909493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7719.913371] Lustre: Skipped 1 previous similar message [ 7719.915436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7719.917937] Lustre: Skipped 2 previous similar messages [ 7725.629154] LustreError: 383702:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7725.631675] LustreError: 383702:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7725.656122] Lustre: server umount lustre-MDT0000 complete [ 7726.882223] LustreError: 381025:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762349 with bad export cookie 5992887278237106229 [ 7726.885189] LustreError: 381025:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7726.885422] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7726.996056] Lustre: server umount lustre-MDT0001 complete [ 7730.405940] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 7731.439323] Key type lgssc unregistered [ 7731.569400] LNet: 384435:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7731.572700] LNetError: 384435:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7731.581300] LNet: Removed LNI 192.168.206.151@tcp [ 7731.862586] Key type .llcrypt unregistered [ 7731.864341] Key type ._llcrypt unregistered [ 7739.281021] Key type ._llcrypt registered [ 7739.282537] Key type .llcrypt registered [ 7739.330845] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 7739.435403] Key type .llcrypt unregistered [ 7739.436532] Key type ._llcrypt unregistered [ 7746.384106] Key type ._llcrypt registered [ 7746.385597] Key type .llcrypt registered [ 7746.428268] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7746.782792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7746.826057] alg: No test for adler32 (adler32-zlib) [ 7747.687354] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 7747.786303] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 7749.376119] Key type lgssc registered [ 7749.769630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7752.526863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7752.533412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7753.698114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7754.987591] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7757.866734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7757.993348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7759.997302] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7761.750816] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7763.106547] LustreError: 386583:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7763.111882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7763.111932] LustreError: 386583:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7763.968299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7764.096599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7765.411213] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7773.664856] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7773.667337] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7773.670852] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7774.691893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7774.692500] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7774.696754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7778.670201] LustreError: 388631:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 7778.709226] Lustre: server umount lustre-OST0000 complete [ 7782.882472] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 7782.884965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7782.888703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7782.890321] Lustre: Skipped 1 previous similar message [ 7784.929243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7784.929729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7785.980587] LustreError: 388833:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7785.983261] LustreError: 388833:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7786.011657] Lustre: server umount lustre-MDT0000 complete [ 7787.244437] LustreError: 386564:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762410 with bad export cookie 4224516903758068387 [ 7787.247985] LustreError: 386564:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7787.248215] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7787.313382] LustreError: 389035:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7787.316261] LustreError: 389035:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7787.371063] Lustre: server umount lustre-MDT0001 complete [ 7791.193147] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 7792.396617] Key type lgssc unregistered [ 7792.530461] LNet: 389566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7792.533667] LNetError: 389566:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7792.542438] LNet: Removed LNI 192.168.206.151@tcp [ 7792.864347] Key type .llcrypt unregistered [ 7792.865512] Key type ._llcrypt unregistered [ 7801.617182] Key type ._llcrypt registered [ 7801.618071] Key type .llcrypt registered [ 7801.655368] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7802.011864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7802.035242] alg: No test for adler32 (adler32-zlib) [ 7802.874630] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 7802.940619] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 7804.512174] Key type lgssc registered [ 7804.868314] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7807.404556] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7807.410161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7822.816277] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7827.936248] LustreError: 390913:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.151@tcp: failed processing log, type 4: rc = -110 [ 7854.560137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7856.418701] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7858.223922] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7860.385715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7860.555702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7860.584989] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7861.831331] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7864.600353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7864.724318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7865.968145] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7867.721036] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7868.927650] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7875.044794] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7875.045135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7875.048059] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7875.052428] Lustre: Skipped 1 previous similar message [ 7875.055732] Lustre: Skipped 1 previous similar message [ 7877.230303] LustreError: 393190:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 7877.278518] Lustre: server umount lustre-OST0000 complete [ 7880.163426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7880.163733] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7880.164693] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7880.177665] Lustre: Skipped 1 previous similar message [ 7884.606292] LustreError: 393392:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 7884.608490] LustreError: 393392:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7884.635196] Lustre: server umount lustre-MDT0000 complete [ 7885.281881] LustreError: 391840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7885.285928] LustreError: 391840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7885.812200] LustreError: 390919:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762508 with bad export cookie 1076854451835018328 [ 7885.814666] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7885.815549] LustreError: 390919:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7885.882260] LustreError: 393594:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7885.884272] LustreError: 393594:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7885.933209] Lustre: server umount lustre-MDT0001 complete [ 7889.383977] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 7890.405441] Key type lgssc unregistered [ 7890.530275] LNet: 394126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7890.532240] LNetError: 394126:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7890.540288] LNet: Removed LNI 192.168.206.151@tcp [ 7890.859651] Key type .llcrypt unregistered [ 7890.860767] Key type ._llcrypt unregistered [ 7899.801682] Key type ._llcrypt registered [ 7899.802731] Key type .llcrypt registered [ 7899.843819] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7900.292296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7900.303909] alg: No test for adler32 (adler32-zlib) [ 7901.157506] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 7901.252494] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 7902.840221] Key type lgssc registered [ 7903.208209] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7905.821973] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7905.827665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7906.983334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7908.263603] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7910.985052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7911.103076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7912.376915] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7914.088863] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7915.244893] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7917.430040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7917.539859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7919.391134] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7921.142185] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7931.881642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7941.996215] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.51@tcp (stopping) [ 7942.115322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7942.115894] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7942.124724] Lustre: Skipped 2 previous similar messages [ 7943.804020] LustreError: 397760:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7943.832888] Lustre: server umount lustre-MDT0000 complete [ 7944.549682] LustreError: 397762:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.51@tcp arrived at 1769762567 with bad export cookie 912149372451142413 [ 7944.556147] LustreError: 397762:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7945.145039] LustreError: 397762:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762568 with bad export cookie 912149372451141762 [ 7945.148898] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7947.233295] LustreError: 395498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7947.233940] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7947.234248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7947.234920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7947.234923] Lustre: Skipped 3 previous similar messages [ 7947.240432] LustreError: 395498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7950.240742] LustreError: 395498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7950.241079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7950.244946] LustreError: 395498:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7951.213259] LustreError: 397963:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7951.215026] LustreError: 397963:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7951.252529] Lustre: server umount lustre-OST0000 complete [ 7953.450941] LustreError: 398317:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7953.453980] LustreError: 398317:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7953.502252] Lustre: server umount lustre-MDT0001 complete [ 7957.208373] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 7958.277405] Key type lgssc unregistered [ 7958.413558] LNet: 398847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7958.416048] LNetError: 398847:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7958.424362] LNet: Removed LNI 192.168.206.151@tcp [ 7958.766281] Key type .llcrypt unregistered [ 7958.767330] Key type ._llcrypt unregistered [ 7968.233135] Key type ._llcrypt registered [ 7968.234407] Key type .llcrypt registered [ 7968.278988] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 7968.738032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7968.756037] alg: No test for adler32 (adler32-zlib) [ 7969.619279] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 7969.705378] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 7971.288178] Key type lgssc registered [ 7971.670402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7974.334098] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7974.342469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7975.514275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7976.815208] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7979.724709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7979.855203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7981.243135] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7983.026664] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7984.298075] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7986.576358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7986.690354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7988.553340] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 7990.312474] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8001.063765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8010.720755] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8010.722929] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8010.725618] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8011.233392] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8011.235251] Lustre: Skipped 1 previous similar message [ 8011.236028] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8012.988509] LustreError: 402483:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8013.012098] Lustre: server umount lustre-MDT0001 complete [ 8016.353490] LustreError: 400930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8016.354380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8016.354506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8016.355646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8016.355651] Lustre: Skipped 1 previous similar message [ 8016.359284] LustreError: 400930:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8020.717096] LustreError: 402686:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8020.719600] LustreError: 402686:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8020.756353] Lustre: server umount lustre-OST0000 complete [ 8021.473354] LustreError: 400930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8022.029680] LustreError: 402888:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8022.031345] LustreError: 402888:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8022.090567] Lustre: server umount lustre-MDT0000 complete [ 8026.712871] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 8027.808635] Key type lgssc unregistered [ 8027.950412] LNet: 403571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8027.952509] LNetError: 403571:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8027.960305] LNet: Removed LNI 192.168.206.151@tcp [ 8028.273730] Key type .llcrypt unregistered [ 8028.274993] Key type ._llcrypt unregistered [ 8035.892539] Key type ._llcrypt registered [ 8035.893289] Key type .llcrypt registered [ 8035.926963] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 8041.390734] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8041.642628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8041.652933] alg: No test for adler32 (adler32-zlib) [ 8042.492110] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 8042.569349] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 8044.144137] Key type lgssc registered [ 8044.478246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8054.887406] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8059.362104] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8059.368045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8060.447261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8060.456898] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8060.490042] Lustre: lustre-MDT0000: new disk, initializing [ 8060.510534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8060.516454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8061.650032] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8067.208582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8067.304583] Lustre: lustre-OST0000: new disk, initializing [ 8067.306968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8067.309660] Lustre: Skipped 1 previous similar message [ 8067.330334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8069.065900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8069.070706] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8069.081640] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8069.326691] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8072.464076] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8077.443991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8077.480951] Lustre: 407503:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8077.508714] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8077.542947] Lustre: lustre-MDT0001: new disk, initializing [ 8077.581229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8077.594121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 8077.596516] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 8078.754434] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8082.014254] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8082.081018] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8083.235118] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8083.302871] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8086.380573] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.51@tcp (stopping) [ 8088.546600] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8088.547034] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8088.547642] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8088.549835] Lustre: Skipped 1 previous similar message [ 8090.811667] LustreError: 410913:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8090.831424] Lustre: server umount lustre-MDT0001 complete [ 8098.786175] LustreError: 407508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8098.790969] LustreError: 407508:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8103.905288] LustreError: 407514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8103.905788] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8103.913272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8103.914684] Lustre: Skipped 1 previous similar message [ 8109.025507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8109.025758] LustreError: 409732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8114.145551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8114.145983] LustreError: 409732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8117.216184] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8117.246053] LustreError: 411185:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 8117.247886] LustreError: 411185:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8117.285025] Lustre: server umount lustre-OST0000 complete [ 8124.735985] LustreError: 411387:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 8124.737843] LustreError: 411387:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8124.798280] Lustre: server umount lustre-MDT0000 complete [ 8129.292123] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 8130.337866] Key type lgssc unregistered [ 8130.463437] LNet: 412070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8130.465699] LNetError: 412070:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8130.476342] LNet: Removed LNI 192.168.206.151@tcp [ 8130.776650] Key type .llcrypt unregistered [ 8130.778122] Key type ._llcrypt unregistered [ 8143.164528] Key type ._llcrypt registered [ 8143.166241] Key type .llcrypt registered [ 8143.209513] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 8143.320371] Key type .llcrypt unregistered [ 8143.321424] Key type ._llcrypt unregistered [ 8149.806308] Key type ._llcrypt registered [ 8149.807756] Key type .llcrypt registered [ 8149.850241] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8150.161326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8150.168421] alg: No test for adler32 (adler32-zlib) [ 8151.021988] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 8151.103783] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 8152.680144] Key type lgssc registered [ 8153.078941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8161.771180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8161.776513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8162.865981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8162.881420] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8162.926942] Lustre: lustre-MDT0000: new disk, initializing [ 8162.952156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8162.959600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8164.253157] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8168.294176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8168.335745] Lustre: 415379:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8168.361073] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8168.363017] Lustre: Skipped 1 previous similar message [ 8168.399396] Lustre: lustre-MDT0001: new disk, initializing [ 8168.433146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8168.448381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8168.451756] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8169.739502] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8172.806070] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8173.948392] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8176.060901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8176.165570] Lustre: lustre-OST0000: new disk, initializing [ 8176.168477] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8176.199896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8177.085486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8177.090080] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8177.128434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8178.025537] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8181.133515] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8183.776875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8183.780361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8183.784106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8187.364570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8187.364795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8187.366821] Lustre: Skipped 1 previous similar message [ 8187.370133] Lustre: Skipped 1 previous similar message [ 8189.507456] LustreError: 417813:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8189.540165] Lustre: server umount lustre-MDT0000 complete [ 8190.803059] LustreError: 415369:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762813 with bad export cookie 14758762320132034980 [ 8190.806300] LustreError: 415369:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8190.806483] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8190.870399] LustreError: 418015:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8190.873665] LustreError: 418015:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8190.922389] Lustre: server umount lustre-MDT0001 complete [ 8202.354201] LustreError: 418215:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8202.356899] LustreError: 418215:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8202.401539] Lustre: server umount lustre-OST0000 complete [ 8205.820894] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 03:47:09 (1769762829) [ 8208.357167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8208.531777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8209.894683] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8212.796381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8212.924209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8214.259184] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8216.052785] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8217.240510] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8230.520922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8230.600831] Lustre: Found index 0 for lustre-OST0000, updating log [ 8230.667785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8232.618657] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8233.767377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 8234.388639] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8237.603199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8237.603272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8237.609499] Lustre: Skipped 1 previous similar message [ 8240.623672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8241.643086] LustreError: 421277:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8241.645031] LustreError: 421277:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8241.679684] Lustre: server umount lustre-OST0000 complete [ 8258.018215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8258.019444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8258.024240] Lustre: Skipped 1 previous similar message [ 8258.026957] Lustre: Skipped 2 previous similar messages [ 8260.347496] LustreError: 421539:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8260.349412] LustreError: 421539:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8260.375640] Lustre: server umount lustre-MDT0000 complete [ 8261.634328] LustreError: 419012:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769762884 with bad export cookie 14758762320132036359 [ 8261.637710] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8261.639516] LustreError: 419012:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8261.760194] Lustre: server umount lustre-MDT0001 complete [ 8264.479305] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 03:48:07 (1769762887) [ 8300.640229] LNet: 1 local NIs in recovery (showing 1): 192.168.206.151@tcp [ 8843.898664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8844.069806] LustreError: 425096:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8844.074781] LustreError: 425096:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8844.097104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8845.339119] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8848.157154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8849.606379] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8851.398693] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8852.555693] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8865.671543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8865.757480] Lustre: Found index 0 for lustre-OST0000, updating log [ 8867.583861] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 8869.263356] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8869.956101] LustreError: 427398:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8869.957832] LustreError: 427398:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8869.991592] Lustre: server umount lustre-OST0000 complete [ 8871.274534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8871.275559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8871.278073] Lustre: Skipped 1 previous similar message [ 8871.281027] Lustre: Skipped 1 previous similar message [ 8877.399884] Lustre: server umount lustre-MDT0000 complete [ 8878.596060] LustreError: 426770:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769763501 with bad export cookie 14758762320132037416 [ 8878.598761] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8878.602204] LustreError: 426770:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8878.710679] Lustre: server umount lustre-MDT0001 complete [ 8895.621291] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 8898.499660] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 8901.728220] LNet: 1 local NIs in recovery (showing 1): 192.168.206.151@tcp [ 8909.360201] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 03:58:52 (1769763532) [ 8914.816883] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 8915.792353] Key type lgssc unregistered [ 8928.224742] LNet: 431350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8928.226680] LNetError: 431350:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8996.838308] LNet: Removed LNI 192.168.206.151@tcp [ 8997.122941] Key type .llcrypt unregistered [ 8997.123930] Key type ._llcrypt unregistered [ 9003.506988] Key type ._llcrypt registered [ 9003.508050] Key type .llcrypt registered [ 9003.549444] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9003.962269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9004.043045] alg: No test for adler32 (adler32-zlib) [ 9004.904611] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 9004.995754] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 9006.576129] Key type lgssc registered [ 9006.987139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9008.528684] LNet: Added LNI 192.168.206.151@tcp100 [8/256/0/180] [ 9008.816627] LNet: Added LNI 192.168.206.151@tcp101 [8/256/0/180] [ 9009.116365] LNet: Added LNI 192.168.206.151@tcp102 [8/256/0/180] [ 9009.421341] LNet: Added LNI 192.168.206.151@tcp103 [8/256/0/180] [ 9009.725985] LNet: Added LNI 192.168.206.151@tcp104 [8/256/0/180] [ 9010.016148] LNet: Added LNI 192.168.206.151@tcp105 [8/256/0/180] [ 9010.280749] LNet: Added LNI 192.168.206.151@tcp106 [8/256/0/180] [ 9010.559324] LNet: Added LNI 192.168.206.151@tcp107 [8/256/0/180] [ 9010.831143] LNet: Added LNI 192.168.206.151@tcp108 [8/256/0/180] [ 9011.125420] LNet: Added LNI 192.168.206.151@tcp109 [8/256/0/180] [ 9011.390940] LNet: Added LNI 192.168.206.151@tcp110 [8/256/0/180] [ 9011.655182] LNet: Added LNI 192.168.206.151@tcp111 [8/256/0/180] [ 9011.919812] LNet: Added LNI 192.168.206.151@tcp112 [8/256/0/180] [ 9012.187966] LNet: Added LNI 192.168.206.151@tcp113 [8/256/0/180] [ 9012.447501] LNet: Added LNI 192.168.206.151@tcp114 [8/256/0/180] [ 9012.709638] LNet: Added LNI 192.168.206.151@tcp115 [8/256/0/180] [ 9012.970843] LNet: Added LNI 192.168.206.151@tcp116 [8/256/0/180] [ 9013.240096] LNet: Added LNI 192.168.206.151@tcp117 [8/256/0/180] [ 9013.520517] LNet: Added LNI 192.168.206.151@tcp118 [8/256/0/180] [ 9013.814068] LNet: Added LNI 192.168.206.151@tcp119 [8/256/0/180] [ 9014.115537] LNet: Added LNI 192.168.206.151@tcp120 [8/256/0/180] [ 9014.400881] LNet: Added LNI 192.168.206.151@tcp121 [8/256/0/180] [ 9014.712425] LNet: Added LNI 192.168.206.151@tcp122 [8/256/0/180] [ 9015.005214] LNet: Added LNI 192.168.206.151@tcp123 [8/256/0/180] [ 9015.285723] LNet: Added LNI 192.168.206.151@tcp124 [8/256/0/180] [ 9015.580479] LNet: Added LNI 192.168.206.151@tcp125 [8/256/0/180] [ 9015.857591] LNet: Added LNI 192.168.206.151@tcp126 [8/256/0/180] [ 9016.167763] LNet: Added LNI 192.168.206.151@tcp127 [8/256/0/180] [ 9016.455724] LNet: Added LNI 192.168.206.151@tcp128 [8/256/0/180] [ 9016.734769] LNet: Added LNI 192.168.206.151@tcp129 [8/256/0/180] [ 9017.045798] LNet: Added LNI 192.168.206.151@tcp130 [8/256/0/180] [ 9017.362108] LNet: Added LNI 192.168.206.151@tcp131 [8/256/0/180] [ 9017.670182] LNet: Added LNI 192.168.206.151@tcp132 [8/256/0/180] [ 9017.964311] LNet: Added LNI 192.168.206.151@tcp133 [8/256/0/180] [ 9018.267392] LNet: Added LNI 192.168.206.151@tcp134 [8/256/0/180] [ 9018.573833] LNet: Added LNI 192.168.206.151@tcp135 [8/256/0/180] [ 9020.693365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9020.698039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9021.746690] Lustre: 434329:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.151@tcp: can't add NID 192.168.206.151@tcp130: rc = -75 [ 9021.807377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9021.817766] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9021.857683] Lustre: lustre-MDT0000: new disk, initializing [ 9021.886672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9021.892803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9023.110143] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9027.115134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9027.165631] Lustre: 434347:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9027.193781] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9027.195803] Lustre: Skipped 1 previous similar message [ 9027.234276] Lustre: lustre-MDT0001: new disk, initializing [ 9027.249022] Lustre: 435127:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.151@tcp: can't add NID 192.168.206.151@tcp130: rc = -75 [ 9027.253764] Lustre: 435127:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 9027.297424] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9027.308456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9027.311439] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9028.460764] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9031.499689] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9032.572438] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9034.664294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9034.765051] Lustre: lustre-OST0000: new disk, initializing [ 9034.766870] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9034.795287] Lustre: 436094:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.151@tcp: can't add NID 192.168.206.151@tcp130: rc = -75 [ 9034.797919] Lustre: 436094:0:(lustre_peer.c:105:class_add_uuid()) Skipped 17 previous similar messages [ 9034.812569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9036.523506] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9039.516095] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9041.724087] LustreError: 436927:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9041.755020] Lustre: server umount lustre-OST0000 complete [ 9042.913874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9042.916352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9042.920775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9044.961534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9044.961997] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9044.979243] Lustre: 437130:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.151@tcp: can't add NID 192.168.206.151@tcp130: rc = -75 [ 9044.981879] Lustre: 437130:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 9050.081261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9050.085544] Lustre: Skipped 2 previous similar messages [ 9054.177391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9054.179263] Lustre: Skipped 1 previous similar message [ 9057.248145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9057.282113] LustreError: 437128:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 9057.284115] LustreError: 437128:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9057.308082] Lustre: server umount lustre-MDT0000 complete [ 9058.534467] LustreError: 435126:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769763681 with bad export cookie 1954251983013811204 [ 9058.537787] LustreError: 435126:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9058.538253] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9058.611953] LustreError: 437334:0:(obd_class.h:479:obd_check_dev()) Device 7 not setup [ 9058.615214] LustreError: 437334:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9058.683330] Lustre: server umount lustre-MDT0001 complete [ 9062.091732] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9063.080822] Key type lgssc unregistered [ 9063.233297] LNet: 437872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9063.235726] LNetError: 437872:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9063.237672] LNet: Removed LNI 192.168.206.151@tcp [ 9064.288102] LNet: Removed LNI 192.168.206.151@tcp100 [ 9064.289383] LNetError: 437872:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9064.292759] LNetError: 437872:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9064.295650] LNet: Removed LNI 192.168.206.151@tcp101 [ 9065.312089] LNet: Removed LNI 192.168.206.151@tcp102 [ 9065.314382] LNetError: 437872:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9065.317583] LNetError: 437872:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9065.319869] LNet: Removed LNI 192.168.206.151@tcp103 [ 9066.336162] LNet: Removed LNI 192.168.206.151@tcp104 [ 9067.360152] LNet: Removed LNI 192.168.206.151@tcp105 [ 9067.362119] LNetError: 437872:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9067.365257] LNetError: 437872:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 9067.368484] LNet: Removed LNI 192.168.206.151@tcp106 [ 9067.369527] LNet: Removed LNI 192.168.206.151@tcp107 [ 9067.370560] LNet: Removed LNI 192.168.206.151@tcp108 [ 9067.371563] LNet: Removed LNI 192.168.206.151@tcp109 [ 9067.372587] LNet: Removed LNI 192.168.206.151@tcp110 [ 9067.373650] LNet: Removed LNI 192.168.206.151@tcp111 [ 9067.374643] LNet: Removed LNI 192.168.206.151@tcp112 [ 9067.375643] LNet: Removed LNI 192.168.206.151@tcp113 [ 9067.376665] LNet: Removed LNI 192.168.206.151@tcp114 [ 9067.377624] LNet: Removed LNI 192.168.206.151@tcp115 [ 9067.378638] LNet: Removed LNI 192.168.206.151@tcp116 [ 9067.379643] LNet: Removed LNI 192.168.206.151@tcp117 [ 9067.380711] LNet: Removed LNI 192.168.206.151@tcp118 [ 9067.381739] LNet: Removed LNI 192.168.206.151@tcp119 [ 9067.382731] LNet: Removed LNI 192.168.206.151@tcp120 [ 9067.383758] LNet: Removed LNI 192.168.206.151@tcp121 [ 9067.384751] LNet: Removed LNI 192.168.206.151@tcp122 [ 9067.385807] LNet: Removed LNI 192.168.206.151@tcp123 [ 9067.386822] LNet: Removed LNI 192.168.206.151@tcp124 [ 9067.387769] LNet: Removed LNI 192.168.206.151@tcp125 [ 9067.388781] LNet: Removed LNI 192.168.206.151@tcp126 [ 9067.389757] LNet: Removed LNI 192.168.206.151@tcp127 [ 9067.390791] LNet: Removed LNI 192.168.206.151@tcp128 [ 9067.391783] LNet: Removed LNI 192.168.206.151@tcp129 [ 9067.392818] LNet: Removed LNI 192.168.206.151@tcp130 [ 9067.393841] LNet: Removed LNI 192.168.206.151@tcp131 [ 9067.394858] LNet: Removed LNI 192.168.206.151@tcp132 [ 9067.395924] LNet: Removed LNI 192.168.206.151@tcp133 [ 9067.396950] LNet: Removed LNI 192.168.206.151@tcp134 [ 9067.404545] LNet: Removed LNI 192.168.206.151@tcp135 [ 9067.685111] Key type .llcrypt unregistered [ 9067.686442] Key type ._llcrypt unregistered [ 9074.699254] Key type ._llcrypt registered [ 9074.700539] Key type .llcrypt registered [ 9074.737760] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 9080.458685] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9080.753863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9080.762464] alg: No test for adler32 (adler32-zlib) [ 9081.608692] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 9081.694542] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 9083.272185] Key type lgssc registered [ 9083.639915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9094.011380] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 04:01:57 (1769763717) [ 9099.453671] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9100.395996] Key type lgssc unregistered [ 9100.514320] LNet: 441884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9100.516274] LNetError: 441884:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9100.528262] LNet: Removed LNI 192.168.206.151@tcp [ 9100.788361] Key type .llcrypt unregistered [ 9100.790026] Key type ._llcrypt unregistered [ 9107.052081] Key type ._llcrypt registered [ 9107.053038] Key type .llcrypt registered [ 9107.090600] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9107.431317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9107.461726] alg: No test for adler32 (adler32-zlib) [ 9108.310455] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 9108.388199] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 9109.960102] Key type lgssc registered [ 9110.304840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9113.199516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9113.205394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9114.295486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9114.306470] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9114.341579] Lustre: lustre-MDT0000: new disk, initializing [ 9114.363992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9114.370423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9115.593189] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9119.497067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9119.537207] Lustre: 443054:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9119.562874] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9119.565539] Lustre: Skipped 1 previous similar message [ 9119.605791] Lustre: lustre-MDT0001: new disk, initializing [ 9119.636734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9119.647495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9119.650450] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9120.789311] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9123.653549] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9124.688631] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9126.806795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9126.904838] Lustre: lustre-OST0000: new disk, initializing [ 9126.907851] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9126.935606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9128.632807] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9128.703730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9128.708688] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9128.740109] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9131.638347] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9132.668177] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing lnet_if_list [ 9134.079754] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 9134.088505] LNet: Added LNI 192.168.206.151@tcp10 [8/256/0/180] [ 9135.671387] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 9135.682249] LNet: Added LNI 192.168.206.151@tcp11 [8/256/0/180] [ 9136.823915] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 9136.834562] LNet: Added LNI 192.168.206.151@tcp12 [8/256/0/180] [ 9137.992843] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 9138.002699] LNet: Added LNI 192.168.206.151@tcp13 [8/256/0/180] [ 9139.198877] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 9139.207865] LNet: Added LNI 192.168.206.151@tcp14 [8/256/0/180] [ 9140.458686] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 9140.467588] LNet: Added LNI 192.168.206.151@tcp15 [8/256/0/180] [ 9141.843846] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 9141.854019] LNet: Added LNI 192.168.206.151@tcp16 [8/256/0/180] [ 9143.227203] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 9143.238876] LNet: Added LNI 192.168.206.151@tcp17 [8/256/0/180] [ 9144.648656] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 9144.658452] LNet: Added LNI 192.168.206.151@tcp18 [8/256/0/180] [ 9146.114197] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 9146.123685] LNet: Added LNI 192.168.206.151@tcp19 [8/256/0/180] [ 9147.541856] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 9147.551752] LNet: Added LNI 192.168.206.151@tcp20 [8/256/0/180] [ 9148.990368] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 9148.998778] LNet: Added LNI 192.168.206.151@tcp21 [8/256/0/180] [ 9150.510429] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 9150.519290] LNet: Added LNI 192.168.206.151@tcp22 [8/256/0/180] [ 9152.012019] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 9152.020921] LNet: Added LNI 192.168.206.151@tcp23 [8/256/0/180] [ 9153.507799] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 9153.517872] LNet: Added LNI 192.168.206.151@tcp24 [8/256/0/180] [ 9155.081669] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 9155.092636] LNet: Added LNI 192.168.206.151@tcp25 [8/256/0/180] [ 9156.731301] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 9156.741381] LNet: Added LNI 192.168.206.151@tcp26 [8/256/0/180] [ 9158.373357] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 9158.384050] LNet: Added LNI 192.168.206.151@tcp27 [8/256/0/180] [ 9160.131247] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 9160.145943] LNet: Added LNI 192.168.206.151@tcp28 [8/256/0/180] [ 9162.116725] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 9162.128238] LNet: Added LNI 192.168.206.151@tcp29 [8/256/0/180] [ 9163.991214] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 9164.001954] LNet: Added LNI 192.168.206.151@tcp30 [8/256/0/180] [ 9165.807474] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 9165.817873] LNet: Added LNI 192.168.206.151@tcp31 [8/256/0/180] [ 9167.618757] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 9167.627355] LNet: Added LNI 192.168.206.151@tcp32 [8/256/0/180] [ 9169.414659] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 9169.425303] LNet: Added LNI 192.168.206.151@tcp33 [8/256/0/180] [ 9171.452062] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 9171.463961] LNet: Added LNI 192.168.206.151@tcp34 [8/256/0/180] [ 9173.491112] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 9173.501982] LNet: Added LNI 192.168.206.151@tcp35 [8/256/0/180] [ 9175.540322] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 9175.554497] LNet: Added LNI 192.168.206.151@tcp36 [8/256/0/180] [ 9177.545606] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 9177.559722] LNet: Added LNI 192.168.206.151@tcp37 [8/256/0/180] [ 9179.583729] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 9179.593234] LNet: Added LNI 192.168.206.151@tcp38 [8/256/0/180] [ 9181.513182] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 9181.523152] LNet: Added LNI 192.168.206.151@tcp39 [8/256/0/180] [ 9183.501560] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 9183.511688] LNet: Added LNI 192.168.206.151@tcp40 [8/256/0/180] [ 9185.456094] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 9185.465262] LNet: Added LNI 192.168.206.151@tcp41 [8/256/0/180] [ 9187.648774] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 9187.659137] LNet: Added LNI 192.168.206.151@tcp42 [8/256/0/180] [ 9189.838437] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 9189.849276] LNet: Added LNI 192.168.206.151@tcp43 [8/256/0/180] [ 9192.044428] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 9192.053944] LNet: Added LNI 192.168.206.151@tcp44 [8/256/0/180] [ 9194.309887] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 9194.322337] LNet: Added LNI 192.168.206.151@tcp45 [8/256/0/180] [ 9196.612903] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 9196.625295] LNet: Added LNI 192.168.206.151@tcp46 [8/256/0/180] [ 9198.791910] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 9198.803317] LNet: Added LNI 192.168.206.151@tcp47 [8/256/0/180] [ 9201.078847] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 9201.088351] LNet: Added LNI 192.168.206.151@tcp48 [8/256/0/180] [ 9203.500213] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 9203.509468] LNet: Added LNI 192.168.206.151@tcp49 [8/256/0/180] [ 9205.954515] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 9205.963989] LNet: Added LNI 192.168.206.151@tcp50 [8/256/0/180] [ 9215.971443] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9215.974300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9215.975084] Lustre: Skipped 2 previous similar messages [ 9215.977989] Lustre: Skipped 1 previous similar message [ 9219.137637] LustreError: 467074:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9219.165127] Lustre: server umount lustre-MDT0000 complete [ 9220.376217] LustreError: 443046:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769763843 with bad export cookie 29841525989330782 [ 9220.379390] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9220.380035] LustreError: 443046:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9220.445937] LustreError: 467276:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9220.447880] LustreError: 467276:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9220.504585] Lustre: server umount lustre-MDT0001 complete [ 9227.450022] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9230.177316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9230.240838] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6a04b5b4265729 [ 9230.243083] Lustre: MGC192.168.206.151@tcp: Connection restored to 0@lo (at 0@lo) [ 9230.350152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9231.546159] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9234.291483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9234.386497] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9234.418778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9235.632741] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9236.453064] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9237.294208] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9237.472148] Lustre: 442328:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769763844/real 1769763844] req@000000004a672884 x1855731774072832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769763860 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9238.388378] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9241.568374] Lustre: 442329:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769763848/real 1769763848] req@000000000eb5b03f x1855731774072960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769763864 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9242.592133] Lustre: 442326:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1769763849/real 1769763849] req@00000000169a0ade x1855731774073088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1769763865 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9245.664912] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9245.667635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9245.672468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9246.693308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9251.441133] LustreError: 469763:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9251.443170] LustreError: 469763:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9251.485068] Lustre: server umount lustre-OST0000 complete [ 9254.880912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9254.883218] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9254.886435] Lustre: Skipped 1 previous similar message [ 9254.887941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9254.889818] Lustre: Skipped 1 previous similar message [ 9262.049694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9262.052243] Lustre: Skipped 3 previous similar messages [ 9267.168097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9267.207798] LustreError: 469964:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9267.209945] LustreError: 469964:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9267.238887] Lustre: server umount lustre-MDT0000 complete [ 9268.412719] LustreError: 443046:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769763891 with bad export cookie 29841525989332777 [ 9268.416027] LustreError: 443046:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9268.416392] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9268.537727] Lustre: server umount lustre-MDT0001 complete [ 9271.803851] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9272.780266] Key type lgssc unregistered [ 9272.935129] LNet: 470706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9272.939228] LNetError: 470706:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9272.941154] LNet: Removed LNI 192.168.206.151@tcp [ 9272.942193] LNet: Removed LNI 192.168.206.151@tcp10 [ 9272.943280] LNet: Removed LNI 192.168.206.151@tcp11 [ 9272.944375] LNet: Removed LNI 192.168.206.151@tcp12 [ 9272.945426] LNet: Removed LNI 192.168.206.151@tcp13 [ 9272.946579] LNet: Removed LNI 192.168.206.151@tcp14 [ 9272.947681] LNet: Removed LNI 192.168.206.151@tcp15 [ 9272.948692] LNet: Removed LNI 192.168.206.151@tcp16 [ 9272.949803] LNet: Removed LNI 192.168.206.151@tcp17 [ 9272.950848] LNet: Removed LNI 192.168.206.151@tcp18 [ 9272.951958] LNet: Removed LNI 192.168.206.151@tcp19 [ 9272.952995] LNet: Removed LNI 192.168.206.151@tcp20 [ 9272.954082] LNet: Removed LNI 192.168.206.151@tcp21 [ 9272.955057] LNet: Removed LNI 192.168.206.151@tcp22 [ 9272.956035] LNet: Removed LNI 192.168.206.151@tcp23 [ 9272.957088] LNet: Removed LNI 192.168.206.151@tcp24 [ 9272.958127] LNet: Removed LNI 192.168.206.151@tcp25 [ 9272.959178] LNet: Removed LNI 192.168.206.151@tcp26 [ 9272.960165] LNet: Removed LNI 192.168.206.151@tcp27 [ 9272.961151] LNet: Removed LNI 192.168.206.151@tcp28 [ 9272.962158] LNet: Removed LNI 192.168.206.151@tcp29 [ 9272.963213] LNet: Removed LNI 192.168.206.151@tcp30 [ 9272.964237] LNet: Removed LNI 192.168.206.151@tcp31 [ 9272.965284] LNet: Removed LNI 192.168.206.151@tcp32 [ 9272.966401] LNet: Removed LNI 192.168.206.151@tcp33 [ 9272.967412] LNet: Removed LNI 192.168.206.151@tcp34 [ 9272.968437] LNet: Removed LNI 192.168.206.151@tcp35 [ 9272.969531] LNet: Removed LNI 192.168.206.151@tcp36 [ 9272.970554] LNet: Removed LNI 192.168.206.151@tcp37 [ 9272.971572] LNet: Removed LNI 192.168.206.151@tcp38 [ 9272.972568] LNet: Removed LNI 192.168.206.151@tcp39 [ 9272.973524] LNet: Removed LNI 192.168.206.151@tcp40 [ 9272.974582] LNet: Removed LNI 192.168.206.151@tcp41 [ 9272.975551] LNet: Removed LNI 192.168.206.151@tcp42 [ 9272.976529] LNet: Removed LNI 192.168.206.151@tcp43 [ 9272.977568] LNet: Removed LNI 192.168.206.151@tcp44 [ 9272.978630] LNet: Removed LNI 192.168.206.151@tcp45 [ 9272.979630] LNet: Removed LNI 192.168.206.151@tcp46 [ 9272.980665] LNet: Removed LNI 192.168.206.151@tcp47 [ 9272.981695] LNet: Removed LNI 192.168.206.151@tcp48 [ 9272.982748] LNet: Removed LNI 192.168.206.151@tcp49 [ 9272.988295] LNet: Removed LNI 192.168.206.151@tcp50 [ 9273.231469] Key type .llcrypt unregistered [ 9273.232674] Key type ._llcrypt unregistered [ 9280.345159] Key type ._llcrypt registered [ 9280.346223] Key type .llcrypt registered [ 9280.383535] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9280.659736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9280.693735] alg: No test for adler32 (adler32-zlib) [ 9281.541100] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 9281.618048] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 9283.192123] Key type lgssc registered [ 9283.526793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9285.925297] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9285.936391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9287.108821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9288.347755] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9291.456583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9291.578132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9292.938885] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9294.764708] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9295.821139] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9297.833202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9297.890313] LustreError: 473518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9297.895078] LustreError: 473518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9297.935318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9299.615689] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9301.171309] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9302.055823] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9310.599035] loop0: detected capacity change from 0 to 204800000 [ 9310.656481] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9310.659477] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9310.662524] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9312.433752] loop0: detected capacity change from 0 to 204800000 [ 9312.488278] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9312.493979] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9312.497689] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9314.033842] loop0: detected capacity change from 0 to 204800000 [ 9315.754363] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9315.805126] Lustre: fs15246-OST0000: new disk, initializing [ 9315.808129] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 9315.824046] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9317.558152] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9321.176993] loop1: detected capacity change from 0 to 204800000 [ 9322.854094] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9322.885850] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 9322.903824] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 9322.937486] Lustre: fs15246-MDT0000: new disk, initializing [ 9322.978763] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9322.986668] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9323.017428] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 9324.065732] Lustre: 476286:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000639718a3 x1855731955429632/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1769763963 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 9324.085619] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9333.152227] LustreError: 476286:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 9333.157962] LustreError: 476286:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 9333.162578] LustreError: 476286:0:(osp_precreate.c:459:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 9335.781514] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9335.787168] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 9336.828097] LustreError: 477406:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 9336.849537] Lustre: server umount fs15246-MDT0000 complete [ 9339.586222] LustreError: 477907:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 9339.589162] LustreError: 477907:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9339.605880] Lustre: server umount fs15246-OST0000 complete [ 9342.945687] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9342.946093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9342.948854] LustreError: Skipped 1 previous similar message [ 9342.952026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9342.955123] Lustre: Skipped 1 previous similar message [ 9346.019192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9346.020791] Lustre: Skipped 1 previous similar message [ 9348.782420] LustreError: 478408:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9348.784263] LustreError: 478408:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 9348.815211] Lustre: server umount lustre-OST0000 complete [ 9351.138518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9351.139803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9351.141521] Lustre: Skipped 2 previous similar messages [ 9356.033900] LustreError: 478610:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9356.035699] LustreError: 478610:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9356.055693] Lustre: server umount lustre-MDT0000 complete [ 9356.256945] LustreError: 472634:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9356.261102] LustreError: 472634:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9357.233517] LustreError: 478322:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769763980 with bad export cookie 90544198949999398 [ 9357.236364] LustreError: 478322:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9357.237681] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9357.357980] Lustre: server umount lustre-MDT0001 complete [ 9360.685530] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9361.712624] Key type lgssc unregistered [ 9361.844282] LNet: 479344:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9361.846244] LNetError: 479344:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9361.855511] LNet: Removed LNI 192.168.206.151@tcp [ 9362.156332] Key type .llcrypt unregistered [ 9362.158084] Key type ._llcrypt unregistered [ 9368.834138] Key type ._llcrypt registered [ 9368.835118] Key type .llcrypt registered [ 9368.872293] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9369.205624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9369.224207] alg: No test for adler32 (adler32-zlib) [ 9370.088727] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 9370.176551] LNet: Added LNI 192.168.206.151@tcp [8/256/0/180] [ 9371.760150] Key type lgssc registered [ 9372.151710] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9383.764791] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 9386.654270] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9397.094066] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9397.101706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9398.184310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9398.194267] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9398.231859] Lustre: lustre-MDT0000: new disk, initializing [ 9398.254265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9398.261364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9399.421892] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9403.263364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9403.300925] Lustre: 483737:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9403.322522] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9403.324145] Lustre: Skipped 1 previous similar message [ 9403.359455] Lustre: lustre-MDT0001: new disk, initializing [ 9403.389553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9403.400321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9403.403036] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9404.605373] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9407.486966] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9408.560378] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9410.595914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9410.687830] Lustre: lustre-OST0000: new disk, initializing [ 9410.690707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9410.715434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9412.403134] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_default_debug -1 all [ 9414.654823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9414.657955] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9414.691203] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9415.314075] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9416.588579] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9416.650123] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9417.823705] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9417.892881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9419.746550] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9419.747364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9419.753780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9419.756815] Lustre: Skipped 1 previous similar message [ 9420.527051] LustreError: 486665:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9420.561321] Lustre: server umount lustre-OST0000 complete [ 9423.840785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9423.842391] LustreError: Skipped 1 previous similar message [ 9423.843193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9423.845556] Lustre: Skipped 1 previous similar message [ 9423.846788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9424.865436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9424.865980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9429.985120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9429.986698] Lustre: Skipped 2 previous similar messages [ 9435.105122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9435.106988] Lustre: Skipped 1 previous similar message [ 9436.128144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9436.166489] LustreError: 486866:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9436.168744] LustreError: 486866:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9436.200534] Lustre: server umount lustre-MDT0000 complete [ 9437.366612] LustreError: 484515:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1769764060 with bad export cookie 15636156661039513398 [ 9437.371112] LustreError: MGC192.168.206.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9437.371947] LustreError: 484515:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9437.436802] LustreError: 487071:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9437.438820] LustreError: 487071:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9437.492452] Lustre: server umount lustre-MDT0001 complete [ 9444.558712] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing set_hostid [ 9447.268351] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing load_modules_local [ 9456.586624] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 9102 sec ========== 04:07:59 (1769764079) [ 9457.081719] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 04:08:00 (1769764080) === [ 9458.088398] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 04:08:01 (1769764081) === [ 9470.112117] Lustre: DEBUG MARKER: oleg651-server.virtnet: executing unload_modules_local [ 9471.033343] Key type lgssc unregistered [ 9471.150379] LNet: 491907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9471.152778] LNetError: 491907:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9471.161308] LNet: Removed LNI 192.168.206.151@tcp [ 9471.441503] Key type .llcrypt unregistered [ 9471.442496] Key type ._llcrypt unregistered