[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1033468836 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002487] x2apic enabled [ 0.003012] Switched APIC routing to physical x2apic. [ 0.004024] kvm-guest: setup PV IPIs [ 0.007843] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008030] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009020] pid_max: default: 32768 minimum: 301 [ 0.010289] LSM: Security Framework initializing [ 0.011070] Yama: becoming mindful. [ 0.012057] SELinux: Initializing. [ 0.013100] *** VALIDATE selinux *** [ 0.023007] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028027] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029184] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030139] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031139] *** VALIDATE tmpfs *** [ 0.033020] *** VALIDATE proc *** [ 0.034274] *** VALIDATE cgroup *** [ 0.035016] *** VALIDATE cgroup2 *** [ 0.037184] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038174] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040048] Spectre V2 : User space: Vulnerable [ 0.041011] Speculative Store Bypass: Vulnerable [ 0.044516] debug: unmapping init [mem 0xffffffffa9259000-0xffffffffa9260fff] [ 0.047245] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048864] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049031] ... version: 2 [ 0.050020] ... bit width: 48 [ 0.051020] ... generic registers: 4 [ 0.052017] ... value mask: 0000ffffffffffff [ 0.053024] ... max period: 00007fffffffffff [ 0.054018] ... fixed-purpose events: 3 [ 0.055016] ... event mask: 000000070000000f [ 0.056318] rcu: Hierarchical SRCU implementation. [ 0.059779] smp: Bringing up secondary CPUs ... [ 0.060697] x86: Booting SMP configuration: [ 0.061037] .... node #0, CPUs: #1 #2 #3 [ 0.078020] smp: Brought up 1 node, 4 CPUs [ 0.080018] smpboot: Max logical packages: 1 [ 0.081021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.171757] node 0 deferred pages initialised in 88ms [ 0.187337] devtmpfs: initialized [ 0.188518] x86/mm: Memory block size: 128MB [ 0.201127] gcov: version magic: 0x41383552 [ 0.220857] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.221000] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.221000] pinctrl core: initialized pinctrl subsystem [ 0.221000] [ 0.221000] ************************************************************* [ 0.221000] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.221000] ** ** [ 0.221000] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.221000] ** ** [ 0.221000] ** This means that this kernel is built to expose internal ** [ 0.221000] ** IOMMU data structures, which may compromise security on ** [ 0.221000] ** your system. ** [ 0.221000] ** ** [ 0.221000] ** If you see this message and you are not debugging the ** [ 0.221000] ** kernel, report this immediately to your vendor! ** [ 0.221000] ** ** [ 0.221000] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.221000] ************************************************************* [ 0.221341] NET: Registered protocol family 16 [ 0.224846] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.232083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.241083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.248053] cpuidle: using governor menu [ 0.252461] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.257024] PCI: Using configuration type 1 for base access [ 0.261145] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.276355] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.281028] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.287145] cryptd: max_cpu_qlen set to 1000 [ 0.292277] ACPI: Added _OSI(Module Device) [ 0.293000] ACPI: Added _OSI(Processor Device) [ 0.294236] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.297254] ACPI: Added _OSI(Processor Aggregator Device) [ 0.305763] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.318094] ACPI: Interpreter enabled [ 0.321079] ACPI: PM: (supports S0 S3 S4 S5) [ 0.324016] ACPI: Using IOAPIC for interrupt routing [ 0.327242] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.333764] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.352125] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.356055] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.362025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.368328] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.377000] acpiphp: Slot [2] registered [ 0.379417] acpiphp: Slot [5] registered [ 0.382396] acpiphp: Slot [6] registered [ 0.385306] acpiphp: Slot [7] registered [ 0.388309] acpiphp: Slot [8] registered [ 0.391236] acpiphp: Slot [9] registered [ 0.394705] acpiphp: Slot [10] registered [ 0.397474] acpiphp: Slot [3] registered [ 0.400334] acpiphp: Slot [4] registered [ 0.403288] acpiphp: Slot [11] registered [ 0.406139] acpiphp: Slot [12] registered [ 0.409735] acpiphp: Slot [13] registered [ 0.412341] acpiphp: Slot [14] registered [ 0.415219] acpiphp: Slot [15] registered [ 0.418225] acpiphp: Slot [16] registered [ 0.422149] acpiphp: Slot [17] registered [ 0.425155] acpiphp: Slot [18] registered [ 0.429217] acpiphp: Slot [19] registered [ 0.432355] acpiphp: Slot [20] registered [ 0.435321] acpiphp: Slot [21] registered [ 0.438212] acpiphp: Slot [22] registered [ 0.441266] acpiphp: Slot [23] registered [ 0.445150] acpiphp: Slot [24] registered [ 0.447485] acpiphp: Slot [25] registered [ 0.451175] acpiphp: Slot [26] registered [ 0.453135] acpiphp: Slot [27] registered [ 0.456139] acpiphp: Slot [28] registered [ 0.459590] acpiphp: Slot [29] registered [ 0.462493] acpiphp: Slot [30] registered [ 0.466228] acpiphp: Slot [31] registered [ 0.469489] PCI host bridge to bus 0000:00 [ 0.473033] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.476029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.481092] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.487031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.492032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.498032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.502202] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.506623] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.512101] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.529027] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.535061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.539023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.543023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.547037] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.550953] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.555186] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.559049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.563141] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.569016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.583018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.589016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.596835] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.605018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.612017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.627017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.639037] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.649021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.659019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.674037] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.692747] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.702020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.709020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.726024] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.739389] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.754102] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.768030] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.826021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.836000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.842015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.847020] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.855016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.865286] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.871016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.877018] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.930018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.944212] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.946425] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.949365] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.952379] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.955220] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.961184] iommu: Default domain type: Passthrough [ 0.962468] SCSI subsystem initialized [ 0.964148] ACPI: bus type USB registered [ 0.965112] usbcore: registered new interface driver usbfs [ 0.966068] usbcore: registered new interface driver hub [ 0.968273] usbcore: registered new device driver usb [ 0.971214] pps_core: LinuxPPS API ver. 1 registered [ 0.973015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.978178] PTP clock support registered [ 0.981122] EDAC MC: Ver: 3.0.0 [ 0.983224] PCI: Using ACPI for IRQ routing [ 0.986838] NetLabel: Initializing [ 0.988020] NetLabel: domain hash size = 128 [ 0.990012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.993142] NetLabel: unlabeled traffic allowed by default [ 0.996183] vgaarb: loaded [ 0.998072] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.000014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.014416] clocksource: Switched to clocksource kvm-clock [ 1.148283] VFS: Disk quotas dquot_6.6.0 [ 1.150444] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.153666] *** VALIDATE ramfs *** [ 1.155243] *** VALIDATE hugetlbfs *** [ 1.157259] pnp: PnP ACPI init [ 1.160151] pnp: PnP ACPI: found 6 devices [ 1.177782] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.181374] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.183970] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.186449] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.189298] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.192160] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.195211] NET: Registered protocol family 2 [ 1.198099] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.203872] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.207917] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.213862] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.217291] TCP: Hash tables configured (established 65536 bind 65536) [ 1.220699] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.224333] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.227748] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.231221] NET: Registered protocol family 1 [ 1.234261] RPC: Registered named UNIX socket transport module. [ 1.237063] RPC: Registered udp transport module. [ 1.238850] RPC: Registered tcp transport module. [ 1.240754] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.243425] NET: Registered protocol family 44 [ 1.245345] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.247881] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.250338] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.253066] PCI: CLS 0 bytes, default 64 [ 1.255736] Unpacking initramfs... [ 2.146368] hrtimer: interrupt took 21097010 ns [ 3.337255] debug: unmapping init [mem 0xffff94b2bcc54000-0xffff94b2bffbffff] [ 3.346594] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.351321] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.360796] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.301860] Initialise system trusted keyrings [ 4.303938] Key type blacklist registered [ 4.307129] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.321323] zbud: loaded [ 4.324559] *** VALIDATE nfs *** [ 4.326036] *** VALIDATE nfs4 *** [ 4.329616] pstore: using deflate compression [ 4.333219] Platform Keyring initialized [ 4.545815] NET: Registered protocol family 38 [ 4.551166] Key type asymmetric registered [ 4.554903] Asymmetric key parser 'x509' registered [ 4.560058] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.569309] io scheduler mq-deadline registered [ 4.571160] io scheduler kyber registered [ 4.577834] io scheduler bfq registered [ 4.582903] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.589423] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.596833] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.605422] ACPI: Power Button [PWRF] [ 4.623206] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.642860] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.667855] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.681143] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.702967] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.747304] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.791624] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.803337] Non-volatile memory driver v1.3 [ 4.808154] Linux agpgart interface v0.103 [ 4.885554] virtio_blk virtio1: [vda] 133944 512-byte logical blocks (68.6 MB/65.4 MiB) [ 4.890814] vda: detected capacity change from 0 to 68579328 [ 4.914160] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.916794] vdb: detected capacity change from 0 to 1073741824 [ 4.940827] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.947966] vdc: detected capacity change from 0 to 2621440000 [ 4.978257] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.983143] vdd: detected capacity change from 0 to 2621440000 [ 5.003336] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.012328] vde: detected capacity change from 0 to 4294967296 [ 5.033379] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.037154] vdf: detected capacity change from 0 to 4294967296 [ 5.044026] libphy: Fixed MDIO Bus: probed [ 5.050176] usbcore: registered new interface driver usbserial_generic [ 5.053251] usbserial: USB Serial support registered for generic [ 5.056285] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.061351] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.063836] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.066980] mousedev: PS/2 mouse device common for all mice [ 5.070684] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.075133] rtc_cmos 00:05: RTC can wake from S4 [ 5.081189] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.082919] rtc_cmos 00:05: registered as rtc0 [ 5.091130] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.094548] intel_pstate: CPU model not supported [ 5.096108] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.106642] hid: raw HID events driver (C) Jiri Kosina [ 5.109699] usbcore: registered new interface driver usbhid [ 5.112371] usbhid: USB HID core driver [ 5.114589] drop_monitor: Initializing network drop monitor service [ 5.117410] Initializing XFRM netlink socket [ 5.208989] NET: Registered protocol family 10 [ 5.260140] Segment Routing with IPv6 [ 5.265577] NET: Registered protocol family 17 [ 5.273349] mpls_gso: MPLS GSO support [ 5.306333] RAS: Correctable Errors collector initialized. [ 5.318124] AVX version of gcm_enc/dec engaged. [ 5.326129] AES CTR mode by8 optimization enabled [ 5.785693] sched_clock: Marking stable (5785618797, 0)->(7107970855, -1322352058) [ 5.789851] registered taskstats version 1 [ 5.793729] Loading compiled-in X.509 certificates [ 5.796822] zswap: loaded using pool lzo/zbud [ 5.850264] Key type big_key registered [ 5.869233] Key type encrypted registered [ 5.885089] ima: No TPM chip found, activating TPM-bypass! [ 5.889930] ima: Allocated hash algorithm: sha1 [ 5.893583] ima: No architecture policies found [ 5.896293] evm: Initialising EVM extended attributes: [ 5.900275] evm: security.selinux [ 5.902662] evm: security.ima [ 5.904532] evm: security.capability [ 5.906950] evm: HMAC attrs: 0x1 [ 5.910756] rtc_cmos 00:05: setting system clock to 2026-04-10 14:38:51 UTC (1775831931) [ 5.920742] debug: unmapping init [mem 0xffffffffaa203000-0xffffffffaa3fffff] [ 5.925371] debug: unmapping init [mem 0xffffffffa8f82000-0xffffffffa9258fff] [ 5.935288] Write protecting the kernel read-only data: 28672k [ 5.939861] debug: unmapping init [mem 0xffffffffa7603000-0xffffffffa77fffff] [ 5.943742] debug: unmapping init [mem 0xffffffffa7f14000-0xffffffffa7ffffff] [ 5.989333] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.009630] systemd[1]: Detected virtualization kvm. [ 6.012316] systemd[1]: Detected architecture x86-64. [ 6.015416] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.050386] systemd[1]: No hostname configured. [ 6.052659] systemd[1]: Set hostname to . [ 6.055039] random: systemd: uninitialized urandom read (16 bytes read) [ 6.057916] systemd[1]: Initializing machine ID from random generator. [ 6.231286] random: ln: uninitialized urandom read (6 bytes read) [ 6.484230] random: systemd: uninitialized urandom read (16 bytes read) [ 6.488837] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 6.498410] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 6.510744] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 8.947552] device-mapper: uevent: version 1.0.3 [ 8.951477] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ 11.231946] random: fast init done [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 12.192272] virtio_net virtio0 ens2: renamed from eth0 [ 12.639711] scsi host0: ata_piix [ 12.688194] scsi host1: ata_piix [ 12.690671] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 12.694226] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 21.463443] random: crng init done [ 21.465469] random: 7 urandom warning(s) missed due to ratelimiting [ 25.455721] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 27.599845] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 32.437319] printk: systemd: 26 output lines suppressed due to ratelimiting [ 34.230851] SELinux: Disabled at runtime. [ 34.382803] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 34.400359] systemd[1]: Detected virtualization kvm. [ 34.404496] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 36.913262] systemd[1]: initrd-switch-root.service: Succeeded. [ 36.927537] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 36.951799] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 36.979534] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 37.002786] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 37.022432] systemd[1]: Starting Journal Service... Starting Journal Service... [ 37.056449] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems...[ 37.525569] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 39.908104] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 41.263406] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 41.365768] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 42.003591] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 42.202741] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit)[ 49.198790] Key type dns_resolver registered [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (13s / no limit)[ 50.074216] NFS: Registering the id_resolver key type [ 50.084733] Key type id_resolver registered [ 50.091514] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg448-server login: [ 110.136238] libcfs: loading out-of-tree module taints kernel. [ 110.159066] Key type ._llcrypt registered [ 110.162208] Key type .llcrypt registered [ 110.227376] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 122.800193] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 123.594724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 123.609469] alg: No test for adler32 (adler32-zlib) [ 124.717475] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 125.208834] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 126.879347] Key type lgssc registered [ 127.996193] Lustre: Echo OBD driver; http://www.lustre.org/ [ 141.355238] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 171.869224] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 181.375190] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 181.412846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 182.635459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 182.672281] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 182.769217] Lustre: lustre-MDT0000: new disk, initializing [ 182.874688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 182.902716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 185.852648] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 195.848362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 195.948510] Lustre: 6504:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 195.979497] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 195.983626] Lustre: Skipped 1 previous similar message [ 196.041215] Lustre: lustre-MDT0001: new disk, initializing [ 196.098835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 196.120598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 196.131599] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 199.192730] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.137676] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 209.961190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 210.128232] Lustre: lustre-OST0000: new disk, initializing [ 210.130620] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 210.171513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 213.787849] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 217.118110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 217.131748] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 217.188618] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 224.671043] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 224.828751] Lustre: lustre-OST0001: new disk, initializing [ 224.833675] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 224.929888] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 225.635527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 225.644283] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 225.735556] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 229.413719] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 239.379607] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 242.307792] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 252.010950] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing check_logdir /tmp/testlogs/ [ 256.217219] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing yml_node [ 259.845675] Lustre: DEBUG MARKER: Client: 2.17.51.75 [ 262.063914] Lustre: DEBUG MARKER: MDS: 2.17.51.75 [ 264.203547] Lustre: DEBUG MARKER: OSS: 2.17.51.75 [ 265.620656] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Apr 10 10:43:09 EDT 2026 [ 279.325961] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 280.543616] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 297.446323] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 297.448295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 297.461163] Lustre: Skipped 2 previous similar messages [ 297.466957] Lustre: Skipped 1 previous similar message [ 300.538580] Lustre: server umount lustre-MDT0000 complete [ 303.240887] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775832228 with bad export cookie 18220764319357732613 [ 303.241349] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 303.248506] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 307.680953] LustreError: 8847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 307.682809] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 307.683651] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 307.683655] Lustre: Skipped 2 previous similar messages [ 307.709267] LustreError: 8847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 307.758015] Lustre: Skipped 2 previous similar messages [ 308.712460] LustreError: 6513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 308.712551] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 308.730792] LustreError: 6513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 308.738175] Lustre: Skipped 2 previous similar messages [ 309.613971] Lustre: server umount lustre-MDT0001 complete [ 312.530866] Lustre: server umount lustre-OST0000 complete [ 315.796315] Lustre: server umount lustre-OST0001 complete [ 321.729932] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 331.196295] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 359.876938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 360.136664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 360.175428] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 360.293658] Lustre: lustre-MDT0000: new disk, initializing [ 360.371532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 360.395203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 363.756060] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 371.268875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 371.343262] Lustre: 14981:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 371.391251] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 371.397390] Lustre: Skipped 1 previous similar message [ 371.459864] Lustre: lustre-MDT0001: new disk, initializing [ 371.527793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 371.562515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 371.569699] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 374.304777] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 379.249184] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 381.385626] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 386.230883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 388.089412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 388.112797] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 388.236484] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 391.426761] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 397.649705] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 400.845479] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 401.017386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 403.794153] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 403.938802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 408.547297] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 408.548101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 408.552658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 408.555674] Lustre: Skipped 1 previous similar message [ 413.669561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 413.675273] Lustre: Skipped 2 previous similar messages [ 419.295174] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 419.392786] Lustre: server umount lustre-OST0000 complete [ 422.881337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 422.890350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 422.897879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 422.903105] Lustre: Skipped 2 previous similar messages [ 427.849504] Lustre: server umount lustre-MDT0000 complete [ 429.027636] LustreError: 14992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 429.044898] LustreError: 14992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 429.948330] LustreError: 14971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775832355 with bad export cookie 18220764319357734146 [ 429.951636] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 429.953718] LustreError: 14971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 430.205932] Lustre: server umount lustre-MDT0001 complete [ 432.638680] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:45:57 (1775832357) [ 433.870257] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 434.896150] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:45:59 (1775832359) [ 438.613573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 438.884502] LustreError: 18983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 438.917466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 438.920431] Lustre: Skipped 1 previous similar message [ 441.116567] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 444.384180] LustreError: 18984:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 445.499139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 447.903066] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 450.927812] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 452.857248] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 456.189786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 459.375215] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 461.989977] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 476.129050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 476.135152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 476.139332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 476.146447] Lustre: Skipped 2 previous similar messages [ 476.149902] Lustre: Skipped 2 previous similar messages [ 480.959847] Lustre: server umount lustre-OST0000 complete [ 486.370744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 486.380551] Lustre: Skipped 1 previous similar message [ 497.119125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 497.215607] Lustre: server umount lustre-MDT0000 complete [ 499.021194] LustreError: 21448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775832424 with bad export cookie 18220764319357735028 [ 499.025429] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 499.027296] LustreError: 21448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 499.185318] Lustre: server umount lustre-MDT0001 complete [ 504.317813] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 505.657220] Key type lgssc unregistered [ 505.804345] LNet: 22182:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 505.809451] LNetError: 22182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 505.822160] LNet: Removed LNI 192.168.204.148@tcp [ 506.271253] Key type .llcrypt unregistered [ 506.274596] Key type ._llcrypt unregistered [ 518.655353] Key type ._llcrypt registered [ 518.657150] Key type .llcrypt registered [ 518.729468] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 519.226576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 519.283146] alg: No test for adler32 (adler32-zlib) [ 520.194380] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 520.347477] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 521.983373] Key type lgssc registered [ 522.678852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 526.975220] loop: module loaded [ 527.002311] loop0: detected capacity change from 0 to 204800000 [ 527.095397] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 527.104184] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 527.249655] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 530.476565] loop0: detected capacity change from 0 to 204800000 [ 530.642202] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 530.649291] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 530.656776] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 533.401590] loop0: detected capacity change from 0 to 204800000 [ 536.291191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 536.305627] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 537.437883] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 537.469261] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 537.514426] Lustre: lustre-MDT0000: new disk, initializing [ 537.556620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 537.568644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 539.699081] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 544.719224] loop1: detected capacity change from 0 to 204800000 [ 547.466375] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 547.603818] Lustre: lustre-OST07c6: new disk, initializing [ 547.607468] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 547.611569] Lustre: Skipped 1 previous similar message [ 547.641592] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 550.333529] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 552.484346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 552.491678] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 552.510501] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 558.229845] Lustre: 24314:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 558.233421] Lustre: 24314:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 558.236680] Lustre: 24314:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 558.240488] Lustre: 24314:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 558.243888] Lustre: 24314:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 558.246652] Lustre: 24314:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 572.319825] Lustre: 24313:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 572.335034] Lustre: 24313:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 572.344199] Lustre: 24313:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 572.349299] Lustre: 24313:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 572.359873] Lustre: 24313:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 572.364525] Lustre: 24313:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 576.659489] Lustre: 24314:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 576.665044] Lustre: 24314:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 576.669600] Lustre: 24314:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 576.672789] Lustre: 24314:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 576.677329] Lustre: 24314:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 576.683346] Lustre: 24314:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 576.688057] Lustre: 24314:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 576.692374] Lustre: 24314:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 576.696411] Lustre: 24314:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 576.700030] Lustre: 24314:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 576.704372] Lustre: 24314:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 576.707521] Lustre: 24314:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 579.829575] Lustre: 25317:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 579.836399] Lustre: 25317:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message [ 579.843273] Lustre: 25317:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 579.850952] Lustre: 25317:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 579.855464] Lustre: 25317:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 579.859645] Lustre: 25317:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 579.863834] Lustre: 25317:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 579.869391] Lustre: 25317:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 579.873392] Lustre: 25317:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 579.877095] Lustre: 25317:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 579.881370] Lustre: 25317:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 579.885901] Lustre: 25317:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 586.798087] Lustre: 25317:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 586.802769] Lustre: 25317:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 3 previous similar messages [ 586.806896] Lustre: 25317:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 586.810825] Lustre: 25317:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 586.814534] Lustre: 25317:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 586.819028] Lustre: 25317:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 586.823246] Lustre: 25317:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 586.828112] Lustre: 25317:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 586.832604] Lustre: 25317:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 586.836835] Lustre: 25317:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 586.841013] Lustre: 25317:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 586.844903] Lustre: 25317:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 596.640691] Lustre: 24313:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 596.650070] Lustre: 24313:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 5 previous similar messages [ 596.654436] Lustre: 24313:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 596.658157] Lustre: 24313:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 596.662369] Lustre: 24313:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 596.665944] Lustre: 24313:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 596.670835] Lustre: 24313:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/2 [ 596.675853] Lustre: 24313:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 596.684298] Lustre: 24313:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 596.688806] Lustre: 24313:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 596.693430] Lustre: 24313:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 596.697251] Lustre: 24313:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 639.456774] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 639.457204] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 639.467447] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 644.450873] Lustre: server umount lustre-OST07c6 complete [ 648.664867] Lustre: server umount lustre-MDT0000 complete [ 655.641932] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 656.883784] Key type lgssc unregistered [ 657.046725] LNet: 27852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 657.050480] LNetError: 27852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 657.057375] LNet: Removed LNI 192.168.204.148@tcp [ 657.401124] Key type .llcrypt unregistered [ 657.403093] Key type ._llcrypt unregistered [ 665.867845] Key type ._llcrypt registered [ 665.869677] Key type .llcrypt registered [ 665.923445] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 666.272908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 666.280860] alg: No test for adler32 (adler32-zlib) [ 667.176212] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 667.279141] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 668.871304] Key type lgssc registered [ 669.357641] Lustre: Echo OBD driver; http://www.lustre.org/ [ 672.917477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 672.932208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 674.175603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 676.137554] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 680.122069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 680.324417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 681.998230] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 684.343255] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 685.902953] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 688.855608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 689.026150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 691.786645] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 694.331092] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 699.303104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 705.504262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 705.508868] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 705.516918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 708.068969] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 708.069099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 708.075489] LustreError: Skipped 1 previous similar message [ 708.085286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 709.600083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 709.604231] Lustre: Skipped 1 previous similar message [ 711.391526] Lustre: server umount lustre-OST0000 complete [ 714.720496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 714.721854] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 714.725756] Lustre: Skipped 1 previous similar message [ 714.734385] Lustre: Skipped 1 previous similar message [ 719.331461] Lustre: server umount lustre-MDT0000 complete [ 719.842732] LustreError: 29077:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 719.852496] LustreError: 29077:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 721.163098] LustreError: 29062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775832646 with bad export cookie 13898023440418785851 [ 721.165690] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 721.169568] LustreError: 29062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 721.354351] Lustre: server umount lustre-MDT0001 complete [ 725.668288] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 726.975044] Key type lgssc unregistered [ 727.125678] LNet: 32459:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.131450] LNetError: 32459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 727.142709] LNet: Removed LNI 192.168.204.148@tcp [ 727.513538] Key type .llcrypt unregistered [ 727.515570] Key type ._llcrypt unregistered [ 736.130782] Key type ._llcrypt registered [ 736.132126] Key type .llcrypt registered [ 736.182098] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 736.655850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 736.671626] alg: No test for adler32 (adler32-zlib) [ 737.552731] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 737.657025] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 739.247165] Key type lgssc registered [ 739.700987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 743.091267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 743.104446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 744.363499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 746.105088] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 749.993451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 750.205737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 751.875638] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 754.323109] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 755.968894] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 758.828930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 759.015231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 761.295820] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 763.593801] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 769.255703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 774.624884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 774.625203] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 774.626876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 774.636401] Lustre: Skipped 1 previous similar message [ 776.232311] Lustre: server umount lustre-OST0000 complete [ 779.744707] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 779.752376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 779.755499] Lustre: Skipped 2 previous similar messages [ 780.024445] Lustre: server umount lustre-MDT0000 complete [ 781.738775] LustreError: 33672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775832707 with bad export cookie 7294378297458948063 [ 781.739260] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 781.743153] LustreError: 33672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 781.878113] Lustre: server umount lustre-MDT0001 complete [ 786.402857] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 787.632137] Key type lgssc unregistered [ 787.782414] LNet: 36866:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.788207] LNetError: 36866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 787.804257] LNet: Removed LNI 192.168.204.148@tcp [ 788.189137] Key type .llcrypt unregistered [ 788.191083] Key type ._llcrypt unregistered [ 798.972529] Key type ._llcrypt registered [ 798.974142] Key type .llcrypt registered [ 799.031565] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 799.526955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 799.592080] alg: No test for adler32 (adler32-zlib) [ 800.486446] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 800.587320] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 802.175160] Key type lgssc registered [ 802.643124] Lustre: Echo OBD driver; http://www.lustre.org/ [ 805.980899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 805.993158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 807.220482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 808.868882] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 812.616467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 812.810763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 814.656554] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 817.015335] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 818.624247] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 821.631664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 821.796616] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 824.299902] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 826.610256] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 832.037888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 837.089628] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 837.094175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 837.096628] Lustre: Skipped 2 previous similar messages [ 837.101440] Lustre: Skipped 1 previous similar message [ 840.353317] Lustre: server umount lustre-MDT0000 complete [ 841.955472] LustreError: 38224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775832767 with bad export cookie 5779743227622251141 [ 841.959445] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 841.964615] LustreError: 38224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 842.208257] LustreError: 38237:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 842.208397] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 842.216894] LustreError: 38237:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 842.226860] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 842.230038] Lustre: Skipped 2 previous similar messages [ 846.240264] LustreError: 38239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 846.240503] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 846.248995] LustreError: 38239:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 846.255225] Lustre: Skipped 1 previous similar message [ 847.431593] LustreError: 40393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 848.117823] Lustre: server umount lustre-MDT0001 complete [ 866.402900] Lustre: server umount lustre-OST0000 complete [ 873.246673] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 874.538731] Key type lgssc unregistered [ 874.685533] LNet: 41747:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.690375] LNetError: 41747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 874.701391] LNet: Removed LNI 192.168.204.148@tcp [ 875.064114] Key type .llcrypt unregistered [ 875.065963] Key type ._llcrypt unregistered [ 885.510128] Key type ._llcrypt registered [ 885.512234] Key type .llcrypt registered [ 885.559558] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 886.035307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 886.048525] alg: No test for adler32 (adler32-zlib) [ 886.923880] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 887.021439] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 888.615186] Key type lgssc registered [ 889.063324] Lustre: Echo OBD driver; http://www.lustre.org/ [ 892.102336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 892.111845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 893.300657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 894.800241] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 898.144727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 898.319865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 899.905337] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 902.006846] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 903.515359] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 906.330294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 906.503594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 908.817935] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 911.103322] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 917.799919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 927.830157] Lustre: lustre-OST0000: Not available for connect from 192.168.204.48@tcp (stopping) [ 928.228177] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 928.229434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 928.239063] LustreError: Skipped 1 previous similar message [ 931.247466] Lustre: server umount lustre-OST0000 complete [ 944.538900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 944.540657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 944.546833] Lustre: Skipped 2 previous similar messages [ 944.553792] Lustre: Skipped 3 previous similar messages [ 947.617585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 947.622091] Lustre: Skipped 1 previous similar message [ 950.703411] Lustre: server umount lustre-MDT0000 complete [ 952.329076] LustreError: 43105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775832877 with bad export cookie 3114182117528383904 [ 952.329443] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 952.334975] LustreError: 43105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 958.531665] Lustre: server umount lustre-MDT0001 complete [ 962.976773] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 964.216920] Key type lgssc unregistered [ 964.373560] LNet: 46497:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.378952] LNetError: 46497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 964.389199] LNet: Removed LNI 192.168.204.148@tcp [ 964.767516] Key type .llcrypt unregistered [ 964.769268] Key type ._llcrypt unregistered [ 975.748110] Key type ._llcrypt registered [ 975.750104] Key type .llcrypt registered [ 975.795832] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 976.207495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 976.259465] alg: No test for adler32 (adler32-zlib) [ 977.171877] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 977.298836] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 978.911424] Key type lgssc registered [ 979.420723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 982.884510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 982.895629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 984.143624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 985.868256] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 989.836796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 990.023175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 991.726692] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 994.022465] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 995.613089] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 999.046270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 999.243234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1002.070525] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1004.795687] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1008.489220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1013.451304] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1014.900591] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1025.712740] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-04-1010h55m50s [ 1026.308448] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1029.090151] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1029.091811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1029.094866] Lustre: Skipped 2 previous similar messages [ 1029.096882] Lustre: Skipped 1 previous similar message [ 1030.112592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1030.115747] Lustre: Skipped 3 previous similar messages [ 1031.759342] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.48@tcp (stopping) [ 1033.304964] Lustre: server umount lustre-MDT0000 complete [ 1034.629427] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1036.258063] LustreError: 48591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1036.265163] LustreError: 48591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1037.284205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1037.351195] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1037.464840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1037.503827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1038.506746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1039.155075] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1040.577533] Lustre: DEBUG MARKER: Wait for df (36039) ... [ 1041.218449] Lustre: DEBUG MARKER: done [ 1042.912312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1042.916648] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1042.922857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1042.926147] Lustre: Skipped 1 previous similar message [ 1043.938555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1044.583084] Lustre: server umount lustre-OST0000 complete [ 1048.309180] Lustre: server umount lustre-MDT0000 complete [ 1049.057110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1049.057693] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1049.058135] LustreError: 47874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1049.074284] Lustre: Skipped 1 previous similar message [ 1049.866576] LustreError: 49490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775832975 with bad export cookie 8510394579162153950 [ 1049.870402] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1049.871271] LustreError: 49490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1056.068191] Lustre: server umount lustre-MDT0001 complete [ 1060.280519] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1061.420689] Key type lgssc unregistered [ 1061.559550] LNet: 52422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.563453] LNetError: 52422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1061.570518] LNet: Removed LNI 192.168.204.148@tcp [ 1061.886655] Key type .llcrypt unregistered [ 1061.888489] Key type ._llcrypt unregistered [ 1078.751544] Key type ._llcrypt registered [ 1078.753090] Key type .llcrypt registered [ 1078.803593] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1079.202587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1079.305330] alg: No test for adler32 (adler32-zlib) [ 1080.190448] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 1080.303144] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1081.903119] Key type lgssc registered [ 1082.363849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1085.649529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1085.660779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1086.805293] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1086.816857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1086.897649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1088.627218] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1092.259858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1092.335344] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1092.348855] Lustre: 54589:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1092.482875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1094.128699] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1096.374218] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1097.832895] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1100.561123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1100.652087] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1100.728145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1103.147924] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1105.504608] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1106.989143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1115.849176] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-04-1010h57m20s [ 1116.572356] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1119.666610] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1120.791297] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1120.812489] LustreError: 54595:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1141.346743] Lustre: lustre-MDT0000: Client d0400cc0-3904-42f1-bf3d-3362610159b6 (at 192.168.204.48@tcp) reconnecting [ 1160.863138] LustreError: 54595:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1161.791487] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1162.791477] Lustre: DEBUG MARKER: done [ 1166.819765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1166.823494] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1166.830446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1168.353814] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1168.354357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1168.357550] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1170.738643] Lustre: server umount lustre-OST0000 complete [ 1173.473137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1173.474600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1173.480133] Lustre: Skipped 1 previous similar message [ 1173.482845] Lustre: Skipped 1 previous similar message [ 1178.799707] Lustre: server umount lustre-MDT0000 complete [ 1180.453779] LustreError: 56222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775833106 with bad export cookie 4735559493845396780 [ 1180.455208] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1180.458721] LustreError: 56222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1180.601424] Lustre: server umount lustre-MDT0001 complete [ 1184.908240] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1186.111375] Key type lgssc unregistered [ 1186.255555] LNet: 58556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1186.259893] LNetError: 58556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1186.269558] LNet: Removed LNI 192.168.204.148@tcp [ 1186.627993] Key type .llcrypt unregistered [ 1186.629403] Key type ._llcrypt unregistered [ 1205.414768] Key type ._llcrypt registered [ 1205.417121] Key type .llcrypt registered [ 1205.463603] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1205.961661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1205.969052] alg: No test for adler32 (adler32-zlib) [ 1206.844685] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 1206.946398] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1208.535175] Key type lgssc registered [ 1208.979941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1210.306317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1210.316434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1211.445363] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1211.456790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1211.534226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1211.973481] Lustre: Failing over lustre-MDT0000 [ 1212.096065] Lustre: server umount lustre-MDT0000 complete [ 1214.750272] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:58:59 (1775833139) [ 1217.475823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.662059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1219.185211] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1222.325818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1222.365900] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1222.378522] Lustre: 61316:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1222.481695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1223.982777] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1226.138824] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1227.697632] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1230.278936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1230.374644] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1230.448975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1231.709654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1232.669986] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1234.767505] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1237.898125] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1242.081393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1242.082805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1242.086456] Lustre: Skipped 2 previous similar messages [ 1242.089013] Lustre: Skipped 1 previous similar message [ 1246.043281] Lustre: server umount lustre-MDT0000 complete [ 1247.464182] LustreError: 61308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775833173 with bad export cookie 468856960812599411 [ 1247.465682] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1247.469331] LustreError: 61308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1247.592680] Lustre: server umount lustre-MDT0001 complete [ 1248.895995] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1250.705835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1253.853476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1257.439792] LustreError: 60238:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008bc2840b x1862096002251648/t0(0) o250->MGC192.168.204.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1257.587794] LustreError: 62936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1257.597737] LustreError: 62936:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1257.628875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1257.661186] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1259.108798] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1262.460124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1262.617021] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1264.202608] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1266.339167] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1267.686891] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1267.780740] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1268.704142] Lustre: 60240:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775833177/real 1775833177] req@00000000d0159cab x1862096002251392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775833193 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1271.904978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1271.905439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1271.908149] Lustre: Skipped 1 previous similar message [ 1271.914478] Lustre: Skipped 1 previous similar message [ 1273.951150] Lustre: 60242:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775833183/real 1775833183] req@00000000b2c7282b x1862096002251520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775833199 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1276.004933] Lustre: server umount lustre-MDT0000 complete [ 1277.411384] LustreError: 62936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1277.419392] LustreError: 62936:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1277.547226] LustreError: 61307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775833203 with bad export cookie 468856960812601679 [ 1277.550956] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1277.555017] LustreError: 61307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1277.670388] Lustre: server umount lustre-MDT0001 complete [ 1280.364704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1284.419843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1287.650163] LustreError: 60238:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000861602a x1862096002270336/t0(0) o250->MGC192.168.204.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1287.813126] LustreError: 65455:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1287.849423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1287.853538] Lustre: Skipped 1 previous similar message [ 1287.883598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1289.635487] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1293.282040] LustreError: 62936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1293.282437] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1293.467538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1295.320927] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1297.639978] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1298.916445] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1299.150583] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1299.936120] Lustre: 60242:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775833208/real 1775833208] req@0000000017747a51 x1862096002270080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775833224 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1304.033344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1304.034418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1304.038386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1304.043971] Lustre: Skipped 1 previous similar message [ 1304.046724] Lustre: Skipped 1 previous similar message [ 1305.055149] Lustre: 60239:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775833213/real 1775833213] req@00000000c517796d x1862096002270208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775833229 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1307.359447] Lustre: server umount lustre-MDT0000 complete [ 1308.128174] LustreError: 67918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1308.135624] LustreError: 67918:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1308.789668] LustreError: 61307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775833234 with bad export cookie 468856960812603051 [ 1308.792037] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1308.794863] LustreError: 61307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1308.918395] Lustre: server umount lustre-MDT0001 complete [ 1311.480818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1312.918562] Lustre: DEBUG MARKER: files compared the same [ 1324.004762] Lustre: server umount lustre-OST0000 complete [ 1330.384549] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1331.602966] Key type lgssc unregistered [ 1331.741654] LNet: 70226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1331.745249] LNetError: 70226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1331.758574] LNet: Removed LNI 192.168.204.148@tcp [ 1332.124422] Key type .llcrypt unregistered [ 1332.125926] Key type ._llcrypt unregistered [ 1342.911028] Key type ._llcrypt registered [ 1342.912731] Key type .llcrypt registered [ 1342.959437] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1343.369028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1343.413489] alg: No test for adler32 (adler32-zlib) [ 1344.298976] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 1344.395138] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1345.983202] Key type lgssc registered [ 1346.439248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1349.744880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1349.754900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1351.004260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1352.697493] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug +malloc all [ 1356.303466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1356.432229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1357.921235] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug +malloc all [ 1359.998786] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1361.435176] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1364.045824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1364.177279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1366.388560] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug +malloc all [ 1368.406564] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1375.460329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1382.879611] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1382.882445] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1382.887673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1385.952044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1385.952462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1385.952629] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1387.656630] Lustre: server umount lustre-OST0000 complete [ 1391.072679] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1391.073907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1391.078728] Lustre: Skipped 1 previous similar message [ 1391.081388] Lustre: Skipped 1 previous similar message [ 1396.192583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1396.195811] Lustre: Skipped 1 previous similar message [ 1401.311619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1401.314405] Lustre: Skipped 1 previous similar message [ 1403.359145] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1403.424599] Lustre: server umount lustre-MDT0000 complete [ 1404.797813] LustreError: 71594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775833330 with bad export cookie 17788180049908192721 [ 1404.799082] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1404.803877] LustreError: 71594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1404.918298] Lustre: server umount lustre-MDT0001 complete [ 1408.876723] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1410.064892] Key type lgssc unregistered [ 1410.205509] LNet: 74813:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1410.209260] LNetError: 74813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1410.224620] LNet: Removed LNI 192.168.204.148@tcp [ 1410.547935] Key type .llcrypt unregistered [ 1410.549568] Key type ._llcrypt unregistered [ 1419.834309] Key type ._llcrypt registered [ 1419.835745] Key type .llcrypt registered [ 1419.872862] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1420.332247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1420.339461] alg: No test for adler32 (adler32-zlib) [ 1421.187345] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 1421.275262] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1422.856343] Key type lgssc registered [ 1423.227587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1426.034362] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1426.043472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1441.503333] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1446.623161] LustreError: 76164:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.148@tcp: failed processing log, type 4: rc = -110 [ 1473.247212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1475.347765] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1477.238707] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1479.870044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1479.920352] Lustre: *** cfs_fail_loc=706, val=0*** [ 1480.066636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1480.096257] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1481.371269] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1484.430337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1484.562062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1485.921849] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1487.780575] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1489.006705] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1489.888082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1489.891025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1489.896223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1490.399886] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1490.402928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1490.407645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1492.961687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1492.964399] Lustre: Skipped 1 previous similar message [ 1498.081381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1498.083813] Lustre: Skipped 1 previous similar message [ 1503.202602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1503.206601] Lustre: Skipped 1 previous similar message [ 1504.223101] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1504.279252] Lustre: server umount lustre-OST0000 complete [ 1508.321325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1508.326745] Lustre: Skipped 1 previous similar message [ 1513.440069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1513.442902] Lustre: Skipped 3 previous similar messages [ 1520.095099] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1520.151521] Lustre: server umount lustre-MDT0000 complete [ 1521.436458] LustreError: 76171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775833447 with bad export cookie 14605844137297411804 [ 1521.439746] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1521.441462] LustreError: 76171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1521.574891] Lustre: server umount lustre-MDT0001 complete [ 1525.439510] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1526.480613] Key type lgssc unregistered [ 1526.611424] LNet: 79429:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1526.614375] LNetError: 79429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1526.622488] LNet: Removed LNI 192.168.204.148@tcp [ 1526.923076] Key type .llcrypt unregistered [ 1526.924949] Key type ._llcrypt unregistered [ 1536.691891] Key type ._llcrypt registered [ 1536.692980] Key type .llcrypt registered [ 1536.732235] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1537.165473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1537.173574] alg: No test for adler32 (adler32-zlib) [ 1538.020039] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 1538.096817] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1539.679157] Key type lgssc registered [ 1540.066914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1542.962736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1542.971971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1545.412428] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1548.259257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1548.454629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1549.829134] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1552.999852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1553.122553] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1554.950970] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1557.685062] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1557.690117] Lustre: MGS: Client 35fc3a9e-8bc1-41d5-879c-ae1cf968e347 (at 0@lo) reconnecting [ 1557.778779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1557.806692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1559.084753] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1568.224287] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1568.226546] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1568.230456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1569.250261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1569.251289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1569.256493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1572.950857] Lustre: server umount lustre-OST0000 complete [ 1574.368308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1574.368421] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1574.368958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1574.368961] Lustre: Skipped 1 previous similar message [ 1574.372530] Lustre: Skipped 1 previous similar message [ 1579.487943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1579.490075] Lustre: Skipped 2 previous similar messages [ 1580.286238] Lustre: server umount lustre-MDT0000 complete [ 1581.508365] Lustre: server umount lustre-MDT0001 complete [ 1582.662967] Lustre: server umount MGS complete [ 1586.334338] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 11:05:11 (1775833511) [ 1595.123829] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 1598.067757] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1609.375675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1610.692748] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1614.610402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1614.705163] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1614.726140] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1614.762404] Lustre: lustre-MDT0001: new disk, initializing [ 1614.791879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1614.801950] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1614.805188] Lustre: 88533:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1614.809059] LustreError: 88533:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1616.127859] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1620.258022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1620.345380] Lustre: lustre-OST0000: new disk, initializing [ 1620.346890] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1620.363957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1622.084357] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1624.223856] Lustre: 89721:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000c0f9cff2 x1862096349332224/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1775833565 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1625.067157] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1627.118929] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1627.122905] Lustre: MGS: Client 0032d603-201e-4ca0-9148-3d2ddde97530 (at 0@lo) reconnecting [ 1627.136701] Lustre: 90214:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1627.143430] LustreError: 88519:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1627.146188] LustreError: 88519:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1627.150820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1627.183633] Lustre: lustre-MDT0000: new disk, initializing [ 1627.230451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1628.256529] Lustre: 90242:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000084e81db5 x1862096349337728/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1775833569 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1628.319508] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1628.403099] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1630.305408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1630.309226] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1630.373387] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1647.583876] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1647.586110] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1647.590301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1652.875199] Lustre: server umount lustre-OST0000 complete [ 1655.265737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1655.266325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1655.268096] LustreError: Skipped 1 previous similar message [ 1655.272763] Lustre: Skipped 2 previous similar messages [ 1660.165695] Lustre: server umount lustre-MDT0000 complete [ 1660.385295] LustreError: 88504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1660.391851] LustreError: 88504:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 1667.639880] Lustre: server umount lustre-MDT0001 complete [ 1668.940504] Lustre: server umount MGS complete [ 1672.653035] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 11:06:37 (1775833597) [ 1677.837018] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1678.787317] Key type lgssc unregistered [ 1678.909308] LNet: 92704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1678.911477] LNetError: 92704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1678.919307] LNet: Removed LNI 192.168.204.148@tcp [ 1679.192503] Key type .llcrypt unregistered [ 1679.193760] Key type ._llcrypt unregistered [ 1685.752959] Key type ._llcrypt registered [ 1685.753912] Key type .llcrypt registered [ 1685.787772] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1686.070647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1686.096066] alg: No test for adler32 (adler32-zlib) [ 1686.940222] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 1687.017408] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1688.599170] Key type lgssc registered [ 1688.951986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1691.447200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1691.452846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1692.587417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1693.709578] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1696.385193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1696.516435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1697.891096] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1699.713076] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1700.986788] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1703.049946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1703.146781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1704.915782] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1706.758435] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1717.479062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1727.969672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1727.970312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1727.970808] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1727.974194] Lustre: Skipped 1 previous similar message [ 1730.956354] Lustre: server umount lustre-OST0000 complete [ 1732.575683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1732.579178] LustreError: Skipped 1 previous similar message [ 1732.581327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1732.586176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1732.587919] Lustre: Skipped 1 previous similar message [ 1738.208024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1738.210398] Lustre: Skipped 3 previous similar messages [ 1738.325570] Lustre: server umount lustre-MDT0000 complete [ 1739.494044] LustreError: 96353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775833665 with bad export cookie 12607725470316117585 [ 1739.496370] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1739.497335] LustreError: 96353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1739.602328] Lustre: server umount lustre-MDT0001 complete [ 1742.772313] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1743.702899] Key type lgssc unregistered [ 1743.824332] LNet: 97085:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1743.827147] LNetError: 97085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1743.833411] LNet: Removed LNI 192.168.204.148@tcp [ 1744.111701] Key type .llcrypt unregistered [ 1744.112879] Key type ._llcrypt unregistered [ 1750.532492] Key type ._llcrypt registered [ 1750.534060] Key type .llcrypt registered [ 1750.572176] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1750.909265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1750.938518] alg: No test for adler32 (adler32-zlib) [ 1751.796750] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 1751.887731] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1753.471158] Key type lgssc registered [ 1753.818594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1755.714732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1755.714742] LustreError: 98128:0:(tgt_mount.c:2501:server_fill_super()) cfs_race id 716 sleeping [ 1755.718230] Lustre: Skipped 1 previous similar message [ 1755.723282] LustreError: 98129:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 waking [ 1755.726308] LustreError: 98128:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 1755.729782] LustreError: 98128:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1755.733438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1755.734422] LustreError: 98128:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1756.883822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1759.386771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1759.472815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1760.692111] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1762.391250] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing lsmod [ 1763.366518] LustreError: 99300:0:(tgt_mount.c:2501:server_fill_super()) cfs_race id 716 sleeping [ 1763.366776] LustreError: 99299:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 waking [ 1763.371167] LustreError: 99300:0:(tgt_mount.c:2501:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 1763.373736] LustreError: 99300:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1763.376563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1763.377963] LustreError: 99300:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1763.516746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1764.831849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1764.834815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1764.840218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1768.928140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1768.929445] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1768.930613] Lustre: Skipped 1 previous similar message [ 1768.936390] Lustre: Skipped 1 previous similar message [ 1769.953396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1769.956664] Lustre: Skipped 3 previous similar messages [ 1770.644599] Lustre: server umount lustre-MDT0000 complete [ 1771.884312] LustreError: 98143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775833697 with bad export cookie 14388455823870956062 [ 1771.888107] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1771.888762] LustreError: 98143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1771.970138] LustreError: 98595:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b821ab06 x1862096573709184/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1772.017414] Lustre: server umount lustre-MDT0001 complete [ 1783.312785] Lustre: server umount lustre-OST0000 complete [ 1786.007498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1786.152947] LustreError: 100367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1786.158879] LustreError: 100367:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1786.174637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1787.422057] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1790.366738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1790.497537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1791.889051] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1793.645685] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1794.878992] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1797.190621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1799.171820] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1800.946179] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1811.621136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1822.180358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1822.181650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1822.186195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1822.188122] LustreError: Skipped 1 previous similar message [ 1822.194162] Lustre: Skipped 1 previous similar message [ 1824.146713] Lustre: server umount lustre-OST0000 complete [ 1826.271694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1826.274479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1826.278871] Lustre: Skipped 1 previous similar message [ 1826.280320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1831.570982] Lustre: server umount lustre-MDT0000 complete [ 1832.416043] LustreError: 100367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1832.422483] LustreError: 100367:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1832.852284] LustreError: 100348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775833758 with bad export cookie 14388455823870956720 [ 1832.855964] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1832.857437] LustreError: 100348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1832.980771] Lustre: server umount lustre-MDT0001 complete [ 1836.659274] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1837.783594] Key type lgssc unregistered [ 1837.909344] LNet: 103565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1837.912064] LNetError: 103565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1837.918325] LNet: Removed LNI 192.168.204.148@tcp [ 1838.234277] Key type .llcrypt unregistered [ 1838.235292] Key type ._llcrypt unregistered [ 1847.695411] Key type ._llcrypt registered [ 1847.697082] Key type .llcrypt registered [ 1847.732912] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1848.036184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1848.044684] alg: No test for adler32 (adler32-zlib) [ 1848.908651] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 1848.989541] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1850.567155] Key type lgssc registered [ 1850.923721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1853.475251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1853.482279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1854.631228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1855.980197] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1859.070185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1859.207300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1860.658069] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1862.760973] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1864.185498] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1866.583412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1866.703240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1868.633496] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1870.468727] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1881.065122] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 1883.004833] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1883.728181] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1886.176934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1886.177760] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1886.179248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1886.185212] Lustre: Skipped 1 previous similar message [ 1890.827575] Lustre: server umount lustre-OST0000 complete [ 1895.391880] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1895.394221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1895.399735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1895.402292] Lustre: Skipped 1 previous similar message [ 1896.416444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1896.417171] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1898.199664] Lustre: server umount lustre-MDT0000 complete [ 1899.525588] LustreError: 104922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775833825 with bad export cookie 10566744995385815172 [ 1899.529610] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1899.530470] LustreError: 104922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1899.638312] Lustre: server umount lustre-MDT0001 complete [ 1903.369661] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1904.352652] Key type lgssc unregistered [ 1904.473329] LNet: 108244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1904.475578] LNetError: 108244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1904.482253] LNet: Removed LNI 192.168.204.148@tcp [ 1904.758638] Key type .llcrypt unregistered [ 1904.759730] Key type ._llcrypt unregistered [ 1911.600554] Key type ._llcrypt registered [ 1911.601479] Key type .llcrypt registered [ 1911.636826] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1912.058277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1912.080459] alg: No test for adler32 (adler32-zlib) [ 1912.927106] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 1913.008692] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1914.583142] Key type lgssc registered [ 1914.920808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1917.568302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1917.574350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1918.718936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1919.999675] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1922.951134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1923.085720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1924.443232] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1926.231485] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1927.423988] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1929.786870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1929.915777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1931.849203] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1933.839922] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1944.294143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 1948.641142] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1948.643164] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1948.646933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1949.665710] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1949.666489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1949.666490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1949.675201] Lustre: Skipped 1 previous similar message [ 1953.106622] Lustre: server umount lustre-OST0000 complete [ 1954.784425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1954.784849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1954.788598] Lustre: Skipped 1 previous similar message [ 1959.903961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1959.907102] Lustre: Skipped 2 previous similar messages [ 1960.539639] Lustre: server umount lustre-MDT0000 complete [ 1961.785127] LustreError: 110142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775833887 with bad export cookie 17643013518569830026 [ 1961.787509] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1961.789316] LustreError: 110142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1961.909624] Lustre: server umount lustre-MDT0001 complete [ 1965.364185] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 1966.292168] Key type lgssc unregistered [ 1966.410367] LNet: 112623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1966.414066] LNetError: 112623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1966.422280] LNet: Removed LNI 192.168.204.148@tcp [ 1966.707541] Key type .llcrypt unregistered [ 1966.708873] Key type ._llcrypt unregistered [ 1975.828304] Key type ._llcrypt registered [ 1975.830014] Key type .llcrypt registered [ 1975.864835] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 1976.162945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1976.177546] alg: No test for adler32 (adler32-zlib) [ 1977.042281] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 1977.128244] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 1978.711193] Key type lgssc registered [ 1979.083228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1981.765250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1981.773060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1982.920068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1984.141458] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1986.757114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1986.867043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1988.059689] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1989.859813] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1991.074866] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1993.359906] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1993.483851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1995.411307] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 1997.230857] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2007.843078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2011.083155] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2012.333064] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2012.334943] Lustre: Skipped 2 previous similar messages [ 2013.783766] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2013.787042] Lustre: Skipped 2 previous similar messages [ 2019.325922] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2019.339260] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2023.756337] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.48@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2023.760582] Lustre: Skipped 2 previous similar messages [ 2026.554076] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.48@tcp 192.168.0.[2,10]@tcp [ 2030.812027] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.48/24@tcp in log lustre-MDT0000 [ 2030.814946] Lustre: Skipped 2 previous similar messages [ 2033.911550] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.48/24@tcp [ 2033.914279] Lustre: Skipped 1 previous similar message [ 2037.727994] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2037.731336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2037.735719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2038.752467] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2038.752877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2038.753925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2038.762694] Lustre: Skipped 1 previous similar message [ 2042.316097] Lustre: server umount lustre-OST0000 complete [ 2043.872401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2043.879121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2048.479875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2048.482323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2048.486594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2049.755428] Lustre: server umount lustre-MDT0000 complete [ 2050.987978] LustreError: 113979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775833976 with bad export cookie 5633344809263720167 [ 2050.990807] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2050.992318] LustreError: 113979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2051.096360] Lustre: server umount lustre-MDT0001 complete [ 2054.601047] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2055.564155] Key type lgssc unregistered [ 2055.691367] LNet: 118697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2055.695140] LNetError: 118697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2055.709308] LNet: Removed LNI 192.168.204.148@tcp [ 2055.974696] Key type .llcrypt unregistered [ 2055.975770] Key type ._llcrypt unregistered [ 2065.564636] Key type ._llcrypt registered [ 2065.565862] Key type .llcrypt registered [ 2065.599407] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2065.888508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2065.955989] alg: No test for adler32 (adler32-zlib) [ 2066.799148] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 2066.871388] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 2068.447129] Key type lgssc registered [ 2068.849258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2071.258413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2071.264656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2072.430811] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2072.435933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2072.439318] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.48/24@tcp [ 2073.627752] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2076.258787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2076.348863] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2076.356990] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.48/24@tcp [ 2076.370965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2077.601466] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2079.309514] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2080.505742] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2082.660927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2082.770855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2084.608288] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2085.864774] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2086.363695] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2090.976911] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2090.977772] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2090.978656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2090.980766] Lustre: Skipped 1 previous similar message [ 2090.985593] LustreError: Skipped 1 previous similar message [ 2092.234315] Lustre: server umount lustre-OST0000 complete [ 2095.442816] Lustre: server umount lustre-MDT0000 complete [ 2096.096325] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2096.096772] LustreError: 120217:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2096.100565] Lustre: Skipped 1 previous similar message [ 2096.107095] LustreError: 120217:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2096.644989] LustreError: 122427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775834022 with bad export cookie 15564913976041292494 [ 2096.647612] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2096.649103] LustreError: 122427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2096.758554] Lustre: server umount lustre-MDT0001 complete [ 2100.341134] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2101.360979] Key type lgssc unregistered [ 2101.485354] LNet: 123448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2101.488281] LNetError: 123448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2101.500281] LNet: Removed LNI 192.168.204.148@tcp [ 2101.780423] Key type .llcrypt unregistered [ 2101.781995] Key type ._llcrypt unregistered [ 2109.703338] Key type ._llcrypt registered [ 2109.704335] Key type .llcrypt registered [ 2109.739060] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 2115.280823] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2115.607109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2115.635694] alg: No test for adler32 (adler32-zlib) [ 2116.494092] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 2116.570543] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 2118.151126] Key type lgssc registered [ 2118.504691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2129.347268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2129.352333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2130.447997] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2130.459472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2130.491131] Lustre: lustre-MDT0000: new disk, initializing [ 2130.512419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2130.518591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2131.741781] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2135.717785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2135.761710] Lustre: 126797:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2135.791628] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2135.793975] Lustre: Skipped 1 previous similar message [ 2135.831036] Lustre: lustre-MDT0001: new disk, initializing [ 2135.864075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2135.877508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2135.881356] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2137.121189] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2140.158717] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2141.262294] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2143.351947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2143.442846] Lustre: lustre-OST0000: new disk, initializing [ 2143.445244] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2143.468440] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2145.215837] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2148.200415] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2149.539554] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2149.877246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2149.880338] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2149.908305] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2150.624642] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2151.927450] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2151.990491] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2154.977718] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2154.977907] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2154.982267] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2154.984890] LustreError: Skipped 1 previous similar message [ 2158.677972] Lustre: server umount lustre-OST0000 complete [ 2160.096414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2160.097039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2160.099923] Lustre: Skipped 2 previous similar messages [ 2160.101463] Lustre: Skipped 1 previous similar message [ 2165.216630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2165.219257] Lustre: Skipped 2 previous similar messages [ 2165.980479] Lustre: server umount lustre-MDT0000 complete [ 2167.161994] LustreError: 127576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775834092 with bad export cookie 4140929837417528674 [ 2167.164247] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2167.167100] LustreError: 127576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2167.273301] Lustre: server umount lustre-MDT0001 complete [ 2169.819148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2169.951404] LustreError: 130611:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2169.956394] LustreError: 130611:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2169.971045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2171.199771] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2173.985296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2174.093281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2175.333636] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2177.023774] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2178.162059] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2180.316633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2182.171334] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2183.877727] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2185.192613] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2185.260641] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2186.477939] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2186.546230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2189.771083] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2189.826538] Lustre: lustre-OST0001: new disk, initializing [ 2189.828360] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2189.860065] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2189.862774] Lustre: Skipped 1 previous similar message [ 2189.888950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2189.892381] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2189.926566] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2191.712975] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2194.902613] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2194.974195] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2196.283255] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2196.359056] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2197.569238] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ccb86ae1000.ost_server_uuid 50 [ 2198.059430] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8ccb86ae1000.ost_server_uuid in FULL state after 0 sec [ 2200.035899] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2200.038719] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2200.043996] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2200.046478] Lustre: Skipped 1 previous similar message [ 2205.155783] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2205.157487] Lustre: Skipped 1 previous similar message [ 2214.367138] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2214.402368] Lustre: server umount lustre-OST0001 complete [ 2215.395077] LustreError: 133638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2215.399797] LustreError: 133638:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2217.439836] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2217.442279] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2217.446423] Lustre: Skipped 1 previous similar message [ 2217.448344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2217.450516] Lustre: Skipped 2 previous similar messages [ 2219.488183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2220.515665] LustreError: 134765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2220.520681] LustreError: 134765:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2221.713449] Lustre: server umount lustre-OST0000 complete [ 2225.633061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2225.635227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2225.639695] Lustre: Skipped 2 previous similar messages [ 2229.023776] Lustre: server umount lustre-MDT0000 complete [ 2230.199698] LustreError: 130592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775834155 with bad export cookie 4140929837417529556 [ 2230.201407] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2230.202498] LustreError: 130592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2230.321089] Lustre: server umount lustre-MDT0001 complete [ 2236.578393] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2237.632513] Key type lgssc unregistered [ 2237.757327] LNet: 136445:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2237.760276] LNetError: 136445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2237.772376] LNet: Removed LNI 192.168.204.148@tcp [ 2238.075465] Key type .llcrypt unregistered [ 2238.076429] Key type ._llcrypt unregistered [ 2250.917976] Key type ._llcrypt registered [ 2250.919301] Key type .llcrypt registered [ 2250.960443] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 2256.855748] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2257.212300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2257.259105] alg: No test for adler32 (adler32-zlib) [ 2258.125682] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 2258.214626] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 2259.791166] Key type lgssc registered [ 2260.211941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2271.745656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2271.754191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2272.854675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2272.868132] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2272.907386] Lustre: lustre-MDT0000: new disk, initializing [ 2272.937101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2272.945561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2274.324862] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2278.710953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2278.759924] Lustre: 140535:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2278.785521] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2278.787871] Lustre: Skipped 1 previous similar message [ 2278.825150] Lustre: lustre-MDT0001: new disk, initializing [ 2278.858971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2278.872112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2278.875621] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2280.223520] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2283.248060] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2284.460483] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2286.810778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2286.927349] Lustre: lustre-OST0000: new disk, initializing [ 2286.929396] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2286.962407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2288.944836] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2289.082235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2289.086738] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2289.123154] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2292.034654] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2294.058492] Lustre: Failing over lustre-OST0000 [ 2294.134693] Lustre: server umount lustre-OST0000 complete [ 2306.949095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2307.052309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2307.056943] Lustre: Skipped 1 previous similar message [ 2307.057810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2307.063827] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2308.440870] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2308.942727] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2310.095073] Lustre: 138764:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775834219/real 1775834219] req@00000000256cf1c6 x1862097104299776/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1775834235 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2310.227334] Lustre: Failing over lustre-MDT0000 [ 2310.327952] Lustre: server umount lustre-MDT0000 complete [ 2312.160943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2312.165220] Lustre: Skipped 1 previous similar message [ 2315.359081] Lustre: 138764:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775834224/real 1775834224] req@000000007f8c18d4 x1862097104300928/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1775834240 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2315.366150] Lustre: 138764:0:(client.c:2479:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2317.279942] LustreError: 141502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2317.285030] LustreError: 141502:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2317.894339] LustreError: 140541:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2321.376093] Lustre: 138762:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775834230/real 1775834230] req@00000000b51a3bcd x1862097104302080/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1775834246 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2322.400533] LustreError: 140547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2322.407722] LustreError: 140547:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2323.371844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2323.414523] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2323.418581] Lustre: 140538:0:(mgc_request.c:1917:mgc_process_log()) MGC192.168.204.148@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2323.500552] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2323.520846] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2323.521545] Lustre: lustre-OST0000: Recovery over after 0:15, of 3 clients 3 recovered and 0 were evicted. [ 2323.525377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2323.562127] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2325.013287] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2327.830508] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2328.552542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2328.554718] Lustre: Skipped 1 previous similar message [ 2328.559941] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 2328.577086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2333.664801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2333.667489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2333.668558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2333.672230] Lustre: Skipped 2 previous similar messages [ 2333.676215] Lustre: Skipped 1 previous similar message [ 2335.637552] Lustre: server umount lustre-OST0000 complete [ 2338.785551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2338.785864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2338.786550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2338.788268] LustreError: Skipped 1 previous similar message [ 2338.793475] Lustre: Skipped 1 previous similar message [ 2343.194687] Lustre: server umount lustre-MDT0000 complete [ 2343.906092] LustreError: 140546:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.912259] LustreError: 140546:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2344.544148] LustreError: 140528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775834270 with bad export cookie 5984669698767508962 [ 2344.546675] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2344.548407] LustreError: 140528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2344.671250] Lustre: server umount lustre-MDT0001 complete [ 2348.358366] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2349.409133] Key type lgssc unregistered [ 2349.532301] LNet: 145767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2349.534869] LNetError: 145767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2349.547263] LNet: Removed LNI 192.168.204.148@tcp [ 2349.857129] Key type .llcrypt unregistered [ 2349.858544] Key type ._llcrypt unregistered [ 2369.808344] Key type ._llcrypt registered [ 2369.809264] Key type .llcrypt registered [ 2369.847262] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2370.333114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2370.363861] alg: No test for adler32 (adler32-zlib) [ 2371.213985] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 2371.290706] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 2372.871156] Key type lgssc registered [ 2373.221810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2375.682250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2375.687639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2376.865636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2378.143856] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2381.146067] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2383.123086] LustreError: 147296:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2383.126536] LustreError: 147296:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2383.129575] LustreError: 147296:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 0, failed: rc = -5 [ 2383.242985] Lustre: server umount lustre-MDT0000 complete [ 2387.808582] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 2390.860704] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2402.326208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2402.412788] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2402.425121] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2402.460645] Lustre: lustre-MDT0000: new disk, initializing [ 2402.482821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2402.489609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2403.687346] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2407.630907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2407.669673] Lustre: 150447:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2407.692694] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2407.695675] Lustre: Skipped 1 previous similar message [ 2407.729857] Lustre: lustre-MDT0001: new disk, initializing [ 2407.757712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2407.767530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2407.770580] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2408.959774] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2411.936102] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2413.051519] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2415.113648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2415.205639] Lustre: lustre-OST0000: new disk, initializing [ 2415.208502] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2415.234296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2416.178943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2416.181707] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2416.212374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2416.897146] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2419.696547] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2421.613871] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2422.202213] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:19:06 (1775834346) [ 2487.775452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2487.775950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2487.780249] Lustre: Skipped 2 previous similar messages [ 2487.783996] Lustre: Skipped 2 previous similar messages [ 2492.895630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2492.898737] Lustre: Skipped 2 previous similar messages [ 2493.094438] Lustre: server umount lustre-MDT0000 complete [ 2494.355426] LustreError: 152186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775834419 with bad export cookie 5287825537678403277 [ 2494.356457] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2494.359514] LustreError: 152186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2498.015534] LustreError: 150454:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.015598] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2498.022021] LustreError: 150454:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2498.026664] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2499.039450] LustreError: 150453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2500.510940] Lustre: server umount lustre-MDT0001 complete [ 2507.919490] Lustre: server umount lustre-OST0000 complete [ 2510.925231] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 2514.340644] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2527.096907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2527.187201] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2527.201872] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2527.242612] Lustre: lustre-MDT0000: new disk, initializing [ 2527.268110] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2527.276219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2528.630830] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2532.879308] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2532.920675] Lustre: 156662:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2532.954884] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2532.954976] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2532.956893] Lustre: Skipped 1 previous similar message [ 2532.960251] Lustre: Skipped 1 previous similar message [ 2533.000089] Lustre: lustre-MDT0001: new disk, initializing [ 2533.045812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2533.049510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2534.333349] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2537.513678] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2538.798137] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2541.174207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2541.275331] Lustre: lustre-OST0000: new disk, initializing [ 2541.277936] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2541.308326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2541.311195] Lustre: Skipped 1 previous similar message [ 2543.219900] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2545.659116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2545.663537] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2545.700772] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2546.413285] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2547.836736] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2547.916050] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2549.351858] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2549.428449] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2550.754378] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2550.754850] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2550.755328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2550.762225] Lustre: Skipped 1 previous similar message [ 2552.142399] Lustre: server umount lustre-OST0000 complete [ 2553.824210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2553.826807] LustreError: Skipped 1 previous similar message [ 2553.829120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2559.575581] Lustre: server umount lustre-MDT0000 complete [ 2560.890354] LustreError: 156654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775834486 with bad export cookie 5287825537678562093 [ 2560.893750] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2560.894734] LustreError: 156654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2561.017597] Lustre: server umount lustre-MDT0001 complete [ 2563.939179] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:21:28 (1775834488) [ 2569.501604] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 2572.742650] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2584.401551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2584.497538] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2584.509228] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2584.547967] Lustre: lustre-MDT0000: new disk, initializing [ 2584.571766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2584.579342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2585.973358] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2590.236523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2590.279826] Lustre: 162970:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2590.377637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2591.709389] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2594.949758] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2596.337331] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2598.744123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2600.926300] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2604.080832] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2605.560438] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2605.595679] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2615.777972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2615.778413] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2615.780459] LustreError: Skipped 1 previous similar message [ 2615.784285] Lustre: Skipped 1 previous similar message [ 2615.786703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2615.789724] Lustre: Skipped 5 previous similar messages [ 2617.558337] Lustre: server umount lustre-OST0000 complete [ 2633.183157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2633.243677] Lustre: server umount lustre-MDT0000 complete [ 2634.529252] LustreError: 162962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775834560 with bad export cookie 5287825537678562975 [ 2634.532909] LustreError: 162962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2634.533233] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2634.654970] Lustre: server umount lustre-MDT0001 complete [ 2637.458580] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:22:42 (1775834562) [ 2642.877547] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 2646.127284] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2657.804096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2657.891120] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2657.905439] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2657.908038] Lustre: Skipped 3 previous similar messages [ 2657.946111] Lustre: lustre-MDT0000: new disk, initializing [ 2657.947854] Lustre: Skipped 2 previous similar messages [ 2657.971425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2657.973810] Lustre: Skipped 2 previous similar messages [ 2657.979362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2657.982725] Lustre: Skipped 2 previous similar messages [ 2659.331461] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2663.623674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2663.669033] Lustre: 168891:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2663.775685] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2665.112316] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2668.139877] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2669.262460] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2671.461074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2673.519071] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2673.974804] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2676.739205] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2679.268061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2679.269909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2679.273480] Lustre: Skipped 2 previous similar messages [ 2679.279257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2679.281298] Lustre: Skipped 7 previous similar messages [ 2684.881325] Lustre: server umount lustre-OST0000 complete [ 2689.506094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2689.508827] LustreError: Skipped 1 previous similar message [ 2692.253452] Lustre: server umount lustre-MDT0000 complete [ 2693.549398] LustreError: 170641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775834619 with bad export cookie 5287825537678564032 [ 2693.552473] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2693.553857] LustreError: 170641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2693.674980] Lustre: server umount lustre-MDT0001 complete [ 2697.614778] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2698.716390] Key type lgssc unregistered [ 2698.857272] LNet: 172363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2698.859792] LNetError: 172363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2698.869531] LNet: Removed LNI 192.168.204.148@tcp [ 2699.193323] Key type .llcrypt unregistered [ 2699.194817] Key type ._llcrypt unregistered [ 2708.081077] Key type ._llcrypt registered [ 2708.082782] Key type .llcrypt registered [ 2708.125981] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2708.469060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2708.520447] alg: No test for adler32 (adler32-zlib) [ 2709.373448] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 2709.460340] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 2711.039096] Key type lgssc registered [ 2711.385514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2713.974859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2713.981698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2715.152095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2716.565247] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2719.464595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2719.613834] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2721.022943] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2722.884748] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2724.125644] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2726.449038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2726.569649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2728.403131] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2730.183714] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2739.878948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2755.555442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2755.558194] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2755.560666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2755.563089] LustreError: Skipped 1 previous similar message [ 2758.554441] Lustre: server umount lustre-OST0000 complete [ 2760.673137] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2760.673239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2760.673945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2760.673950] Lustre: Skipped 1 previous similar message [ 2760.684560] Lustre: Skipped 2 previous similar messages [ 2765.792659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2765.795706] Lustre: Skipped 2 previous similar messages [ 2766.043873] Lustre: server umount lustre-MDT0000 complete [ 2767.364396] LustreError: 175381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775834692 with bad export cookie 1652599346101370029 [ 2767.368935] LustreError: 175381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2767.369763] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2767.492853] Lustre: server umount lustre-MDT0001 complete [ 2771.245276] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 2772.230899] Key type lgssc unregistered [ 2772.355419] LNet: 176965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2772.357380] LNetError: 176965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2772.369287] LNet: Removed LNI 192.168.204.148@tcp [ 2772.658282] Key type .llcrypt unregistered [ 2772.659248] Key type ._llcrypt unregistered [ 2781.746437] Key type ._llcrypt registered [ 2781.747617] Key type .llcrypt registered [ 2781.783384] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 2782.293407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2782.313430] alg: No test for adler32 (adler32-zlib) [ 2783.169798] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 2783.256620] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 2784.847207] Key type lgssc registered [ 2785.257724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2787.913715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2787.920180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2789.075622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2790.398483] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2793.276109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2793.411610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2794.832537] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2796.717957] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2797.996889] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2800.254147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2800.364555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2802.322062] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2804.113198] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2813.670336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2819.042961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2819.043482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2819.044642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2819.048900] Lustre: Skipped 1 previous similar message [ 2819.052303] LustreError: Skipped 1 previous similar message [ 2819.663596] Lustre: lustre-OST0000: Not available for connect from 192.168.204.48@tcp (stopping) [ 2819.667020] Lustre: Skipped 1 previous similar message [ 2824.164325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2824.166522] Lustre: Skipped 1 previous similar message [ 2829.282498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2829.284690] Lustre: Skipped 2 previous similar messages [ 2831.327113] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2831.383270] Lustre: server umount lustre-OST0000 complete [ 2833.256976] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8ccb862ad800.ost_server_uuid 50 [ 2833.797256] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8ccb862ad800.ost_server_uuid in DISCONN state after 0 sec [ 2834.353398] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2844.412314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2844.412957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2844.419387] Lustre: Skipped 1 previous similar message [ 2844.421949] Lustre: Skipped 1 previous similar message [ 2850.580087] Lustre: server umount lustre-MDT0000 complete [ 2851.812143] LustreError: 178323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775834777 with bad export cookie 1242313256421556366 [ 2851.815035] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2851.816270] LustreError: 178323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2851.934909] Lustre: server umount lustre-MDT0001 complete [ 2854.918898] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:26:19 (1775834779) [ 2857.207483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2857.344256] LustreError: 181868:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2857.350926] LustreError: 181868:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2857.367153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2858.667150] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2861.470088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2861.577470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2862.795202] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2864.512097] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2865.641496] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2867.708615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2869.541184] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2871.211378] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2873.379889] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2873.461446] Lustre: lustre-OST0001: new disk, initializing [ 2873.463596] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2873.495115] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2873.496957] Lustre: Skipped 1 previous similar message [ 2874.688499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2874.692378] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2874.727879] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2875.347638] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2878.381866] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2879.971280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2879.974980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2879.976563] Lustre: Skipped 3 previous similar messages [ 2885.305902] Lustre: server umount lustre-OST0000 complete [ 2887.099095] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2887.169503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2888.165061] LustreError: 183463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2888.172330] LustreError: 183463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2888.457411] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2888.533753] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2893.280140] LustreError: 183462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2893.284474] LustreError: 183462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2898.401073] LustreError: 183461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2898.407128] LustreError: 183461:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2900.448096] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2900.448539] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2900.450533] LustreError: Skipped 1 previous similar message [ 2900.456091] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2900.460060] Lustre: Skipped 4 previous similar messages [ 2903.520239] LustreError: 183461:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2903.525375] LustreError: 183461:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2904.597076] Lustre: server umount lustre-OST0001 complete [ 2907.615912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2907.618507] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2907.622499] Lustre: Skipped 1 previous similar message [ 2911.963429] Lustre: server umount lustre-MDT0000 complete [ 2913.162961] LustreError: 181849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775834838 with bad export cookie 1242313256421557423 [ 2913.166160] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2913.166393] LustreError: 181849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2913.284857] Lustre: server umount lustre-MDT0001 complete [ 2920.948434] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:27:25 (1775834845) [ 2923.041740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2923.126952] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2923.138361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2923.201941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2924.439363] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2927.220950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.252625] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2927.263776] Lustre: 187640:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2928.581532] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2930.333194] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2931.517513] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2933.697586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2933.774470] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2935.658524] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2937.289447] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2938.894416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2939.456035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2939.479596] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2941.377680] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 2943.050992] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2943.967955] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2943.968033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2943.970149] LustreError: Skipped 1 previous similar message [ 2943.973366] Lustre: Skipped 1 previous similar message [ 2943.975695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2943.978714] Lustre: Skipped 6 previous similar messages [ 2945.541344] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 2958.303117] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2958.343198] Lustre: server umount lustre-OST0000 complete [ 2958.919740] LustreError: 189262:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2971.108024] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2971.111035] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2975.131789] Lustre: server umount lustre-OST0001 complete [ 2982.623669] Lustre: server umount lustre-MDT0000 complete [ 2983.870055] LustreError: 187630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775834909 with bad export cookie 1242313256421558557 [ 2983.873231] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2983.873542] LustreError: 187630:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2983.990549] Lustre: server umount lustre-MDT0001 complete [ 2991.635418] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 11:28:36 (1775834916) [ 2996.829884] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 3000.022152] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 3011.886235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3011.968683] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3011.979279] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3012.015549] Lustre: lustre-MDT0000: new disk, initializing [ 3012.037715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3012.039965] Lustre: Skipped 3 previous similar messages [ 3012.045549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3013.331370] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3017.543175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3017.588558] Lustre: 195214:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3017.616404] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3017.618414] Lustre: Skipped 1 previous similar message [ 3017.660885] Lustre: lustre-MDT0001: new disk, initializing [ 3017.715930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3017.719318] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3019.070752] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3022.183937] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3023.366358] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3025.649497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3025.781217] Lustre: lustre-OST0000: new disk, initializing [ 3025.782947] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3026.616100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3026.619325] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3026.658832] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3027.755160] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3030.837733] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3032.277857] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3032.349887] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3033.745659] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3033.822266] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3036.561430] Lustre: server umount lustre-OST0000 complete [ 3038.175892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3038.178609] LustreError: Skipped 2 previous similar messages [ 3038.180049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3038.183692] Lustre: Skipped 3 previous similar messages [ 3038.185372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3038.187029] Lustre: Skipped 15 previous similar messages [ 3048.445471] Lustre: server umount lustre-MDT0000 complete [ 3049.702164] LustreError: 195206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775834975 with bad export cookie 1242313256421560293 [ 3049.705339] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3049.707175] LustreError: 195206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3060.244476] Lustre: server umount lustre-MDT0001 complete [ 3063.024451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3063.167116] LustreError: 199023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3063.172888] LustreError: 199023:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3064.413985] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3067.232586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3068.682804] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3070.527413] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3071.758940] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3074.093870] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3076.176282] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3077.982428] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3079.463655] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3087.716510] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 3088.974162] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3089.045765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3092.965399] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3092.969855] LustreError: Skipped 1 previous similar message [ 3095.828867] Lustre: server umount lustre-OST0000 complete [ 3097.744848] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3097.818321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3099.154321] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3122.794179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3122.906538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3122.910208] Lustre: Skipped 5 previous similar messages [ 3124.719124] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3126.554666] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3127.122198] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3130.250883] Lustre: server umount lustre-OST0000 complete [ 3131.537740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3131.544821] Lustre: Skipped 4 previous similar messages [ 3133.658083] Lustre: server umount lustre-MDT0000 complete [ 3134.916290] LustreError: 199005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775835060 with bad export cookie 1242313256421561161 [ 3134.918758] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3134.921124] LustreError: 199005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3135.058691] Lustre: server umount lustre-MDT0001 complete [ 3138.042531] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:31:02 (1775835062) [ 3140.297975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3140.458215] LustreError: 204535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3141.838785] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3144.648558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3146.109682] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3147.984176] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3149.213384] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3151.503940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3153.516251] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3155.221579] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3156.569660] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3166.852248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3168.133380] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3168.209192] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3170.438282] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3170.501104] Lustre: lustre-OST0001: new disk, initializing [ 3170.503983] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3172.356536] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3175.403599] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3175.934859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3175.938361] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3175.963117] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3176.752484] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3176.822664] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3178.128478] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3178.198130] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3181.025042] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3181.031770] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3181.034581] Lustre: Skipped 11 previous similar messages [ 3184.957408] Lustre: server umount lustre-OST0001 complete [ 3186.859814] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3186.930517] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3188.237650] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3188.313901] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3206.625818] LustreError: 206130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3206.631694] LustreError: 206130:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 3211.925174] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3213.847748] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3215.577853] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3216.191143] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3222.965209] Lustre: server umount lustre-OST0001 complete [ 3241.104773] Lustre: server umount lustre-OST0000 complete [ 3248.551805] Lustre: server umount lustre-MDT0000 complete [ 3249.773465] LustreError: 204517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775835175 with bad export cookie 1242313256421562078 [ 3249.776663] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3249.778668] LustreError: 204517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3255.982624] Lustre: server umount lustre-MDT0001 complete [ 3263.762158] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:33:08 (1775835188) [ 3265.952875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3266.076790] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3266.088439] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3266.160951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3266.163051] Lustre: Skipped 5 previous similar messages [ 3267.436756] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3270.281758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3270.313482] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3270.323894] Lustre: 212542:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3271.777209] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3273.651173] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3274.883793] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3277.151837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3277.234243] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3279.171176] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3280.935403] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3284.703607] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3286.739941] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3288.552705] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3289.100734] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3290.530387] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3290.606829] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3291.937747] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3292.010209] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3293.136915] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8ccb85a08800.ost_server_uuid 50 [ 3293.634101] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8ccb85a08800.ost_server_uuid in FULL state after 0 sec [ 3293.968311] Lustre: Permanently deactivating lustre-OST0001 [ 3293.974311] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3294.748532] Lustre: Permanently reactivating lustre-OST0001 [ 3296.230898] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3296.234133] LustreError: Skipped 3 previous similar messages [ 3296.235877] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3296.240275] Lustre: Skipped 6 previous similar messages [ 3301.819520] Lustre: server umount lustre-OST0001 complete [ 3309.205361] Lustre: server umount lustre-OST0000 complete [ 3316.633872] Lustre: server umount lustre-MDT0000 complete [ 3317.831330] LustreError: 215072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775835243 with bad export cookie 1242313256421563338 [ 3317.834410] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3317.835319] LustreError: 215072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3317.949358] Lustre: server umount lustre-MDT0001 complete [ 3325.653720] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:34:10 (1775835250) [ 3328.187618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3328.283358] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3328.293995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3328.298979] Lustre: Skipped 5 previous similar messages [ 3329.704647] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3332.536647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3332.569690] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3332.572316] Lustre: Skipped 1 previous similar message [ 3332.582221] Lustre: 218642:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3333.930591] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3335.723149] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3336.930591] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3339.262036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3339.355121] Lustre: Permanently deactivating lustre-OST0000 [ 3339.359897] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3341.299691] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3343.154876] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3343.179955] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3345.580409] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3347.555358] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3348.502288] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3349.328068] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3349.907172] Lustre: Permanently reactivating lustre-OST0000 [ 3357.686222] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3357.689050] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3357.692553] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3357.708714] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3357.714315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3364.992441] Lustre: server umount lustre-OST0001 complete [ 3368.930562] LustreError: 220260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3368.935856] LustreError: 220260:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 3372.303490] Lustre: server umount lustre-OST0000 complete [ 3373.795366] Lustre: server umount lustre-MDT0000 complete [ 3375.092491] LustreError: 218633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775835300 with bad export cookie 1242313256421564976 [ 3375.096286] LustreError: 218633:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3375.096756] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3381.322538] Lustre: server umount lustre-MDT0001 complete [ 3385.149839] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 3386.253923] Key type lgssc unregistered [ 3386.375350] LNet: 223022:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3386.379977] LNetError: 223022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3386.391337] LNet: Removed LNI 192.168.204.148@tcp [ 3386.721769] Key type .llcrypt unregistered [ 3386.723278] Key type ._llcrypt unregistered [ 3395.495629] Key type ._llcrypt registered [ 3395.496929] Key type .llcrypt registered [ 3395.542969] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 3396.038712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3396.052956] alg: No test for adler32 (adler32-zlib) [ 3396.925642] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 3397.017670] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 3398.607198] Key type lgssc registered [ 3399.027930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3402.285291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3402.291643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3403.455963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3404.699163] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3407.507725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3407.546167] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3407.556326] Lustre: 224439:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3407.560917] Lustre: Permanently deactivating lustre-MDT0001 [ 3407.566771] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3407.675302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3408.960340] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3410.778542] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3412.036768] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3412.461347] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3412.463897] LustreError: 224479:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3412.468275] LustreError: 224479:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3412.472737] LustreError: 224479:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 1, failed: rc = -108 [ 3414.312958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3414.432645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3416.393955] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3417.507931] LustreError: 226061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3417.516102] LustreError: 226061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3418.265816] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3420.614548] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3420.694341] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3421.813907] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3422.640746] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3424.527706] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3428.973472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3428.973894] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3430.400948] Lustre: Permanently reactivating lustre-MDT0001 [ 3430.407890] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3430.410964] Lustre: Skipped 2 previous similar messages [ 3432.653619] Lustre: lustre-MDT0001: Client 875c6d0b-c657-4bc4-98c6-ec6f819be4a9 (at 192.168.204.48@tcp) reconnecting [ 3434.704405] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3435.256904] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3437.034648] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3437.123825] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3438.445517] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3439.534123] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [ 3440.328839] Lustre: Permanently deactivating lustre-MDT0001 [ 3440.334600] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3440.337884] Lustre: Skipped 2 previous similar messages [ 3447.296722] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3453.919919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3453.922282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3453.926742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3454.944443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3454.946429] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3454.948780] Lustre: Skipped 1 previous similar message [ 3454.952933] Lustre: Skipped 1 previous similar message [ 3459.619076] Lustre: server umount lustre-MDT0000 complete [ 3460.064895] LustreError: 224443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3460.069665] LustreError: 224443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3460.884153] LustreError: 224430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775835386 with bad export cookie 15523058761714475744 [ 3460.887628] LustreError: 224430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3460.887687] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3461.003113] Lustre: server umount lustre-MDT0001 complete [ 3468.346581] Lustre: server umount lustre-OST0000 complete [ 3472.015964] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:36:36 (1775835396) [ 3475.588712] Lustre: server umount lustre-OST0001 complete [ 3477.656581] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 3480.697954] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 3492.375206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3492.472521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3492.476438] Lustre: Skipped 2 previous similar messages [ 3492.485482] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3492.526498] Lustre: lustre-MDT0000: new disk, initializing [ 3492.550742] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3492.558729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3493.886586] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3498.106275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3498.151914] Lustre: 232186:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3498.173399] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3498.175364] Lustre: Skipped 1 previous similar message [ 3498.211885] Lustre: lustre-MDT0001: new disk, initializing [ 3498.253925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3498.259042] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3499.600190] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3502.743716] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3503.938296] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3506.218148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3506.317724] Lustre: lustre-OST0000: new disk, initializing [ 3506.320355] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3506.346625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3506.349192] Lustre: Skipped 1 previous similar message [ 3508.088651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3508.092911] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3508.126382] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3508.312103] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3511.453987] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3512.147788] LustreError: 232195:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3516.417408] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3516.479030] Lustre: lustre-OST0001: new disk, initializing [ 3516.481051] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3518.434623] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3521.586130] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3522.182088] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3522.239062] LustreError: 232195:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3522.249278] LustreError: 234937:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3522.551266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3522.554761] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3522.580298] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3532.343113] LustreError: 234937:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3536.462443] Lustre: lustre-OST0001: Not available for connect from 192.168.204.48@tcp (stopping) [ 3536.465153] Lustre: Skipped 2 previous similar messages [ 3537.890548] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3537.892651] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3537.894542] Lustre: Skipped 2 previous similar messages [ 3537.901022] LustreError: Skipped 1 previous similar message [ 3539.323054] Lustre: server umount lustre-OST0001 complete [ 3543.010115] LustreError: 234218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3543.015792] LustreError: 234218:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3548.131818] LustreError: 233911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3548.139444] LustreError: 233911:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3553.251976] LustreError: 233911:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3553.252701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3553.255156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3553.255165] Lustre: Skipped 2 previous similar messages [ 3553.257404] LustreError: 233911:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3553.262627] Lustre: Skipped 1 previous similar message [ 3557.450527] Lustre: server umount lustre-OST0000 complete [ 3559.904246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3559.906813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3559.910830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3559.912574] Lustre: Skipped 1 previous similar message [ 3564.975690] Lustre: server umount lustre-MDT0000 complete [ 3566.255202] LustreError: 232964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775835491 with bad export cookie 15523058761714478313 [ 3566.257948] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3566.259341] LustreError: 232964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3572.406639] Lustre: server umount lustre-MDT0001 complete [ 3576.108817] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 3577.131818] Key type lgssc unregistered [ 3577.256283] LNet: 236850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3577.259139] LNetError: 236850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3577.265312] LNet: Removed LNI 192.168.204.148@tcp [ 3577.550651] Key type .llcrypt unregistered [ 3577.551964] Key type ._llcrypt unregistered [ 3591.679694] Key type ._llcrypt registered [ 3591.680966] Key type .llcrypt registered [ 3591.715954] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 3592.025925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3592.086804] alg: No test for adler32 (adler32-zlib) [ 3592.936825] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 3593.014327] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 3594.599221] Key type lgssc registered [ 3594.972594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3597.566091] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3597.576398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3598.699149] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3598.709607] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3598.797461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3600.324308] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3603.234600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3603.267204] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3603.280652] Lustre: 239016:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3603.383144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3604.669292] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3606.488929] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3607.671347] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3609.913009] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3609.981870] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3610.055587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3611.892655] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3613.332850] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3613.654605] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3654.627078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3654.627576] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3654.628165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3654.631060] Lustre: Skipped 1 previous similar message [ 3654.634441] LustreError: Skipped 1 previous similar message [ 3658.389698] Lustre: server umount lustre-OST0000 complete [ 3659.918613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3664.038673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3664.162925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3666.022711] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3667.819324] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3675.428220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3675.434431] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3675.434436] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3675.440106] Lustre: Skipped 1 previous similar message [ 3701.217508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3701.218967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3701.219572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3701.219577] Lustre: Skipped 1 previous similar message [ 3701.221824] Lustre: Skipped 1 previous similar message [ 3701.225900] LustreError: Skipped 1 previous similar message [ 3706.337972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3706.340230] Lustre: Skipped 2 previous similar messages [ 3707.099745] Lustre: server umount lustre-OST0000 complete [ 3710.944171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3710.946378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3710.950301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3714.458803] Lustre: server umount lustre-MDT0000 complete [ 3715.710198] LustreError: 239008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775835641 with bad export cookie 2294240658259783302 [ 3715.713605] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3715.717081] LustreError: 239008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3715.827787] Lustre: server umount lustre-MDT0001 complete [ 3719.353876] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 3720.389907] Key type lgssc unregistered [ 3720.512501] LNet: 243965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3720.514923] LNetError: 243965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3720.522407] LNet: Removed LNI 192.168.204.148@tcp [ 3720.798282] Key type .llcrypt unregistered [ 3720.799459] Key type ._llcrypt unregistered [ 3730.144190] Key type ._llcrypt registered [ 3730.145380] Key type .llcrypt registered [ 3730.187934] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 3730.512788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3730.539077] alg: No test for adler32 (adler32-zlib) [ 3731.399622] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 3731.488399] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 3733.071174] Key type lgssc registered [ 3733.431113] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3736.230657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3736.239217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3737.405458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3738.799785] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3742.628133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3742.744089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3744.191413] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3746.205942] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3747.578072] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3750.099832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3750.210030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3752.062026] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3753.902371] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3761.513430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3771.873427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3771.874665] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3771.874971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3771.877115] Lustre: Skipped 1 previous similar message [ 3776.992630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3776.995901] Lustre: Skipped 2 previous similar messages [ 3777.809157] Lustre: server umount lustre-OST0000 complete [ 3782.113051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3782.113834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3782.116660] Lustre: Skipped 1 previous similar message [ 3782.121235] Lustre: Skipped 1 previous similar message [ 3787.232615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3787.235544] Lustre: Skipped 1 previous similar message [ 3792.352444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3792.355774] Lustre: Skipped 1 previous similar message [ 3793.375095] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3793.437967] Lustre: server umount lustre-MDT0000 complete [ 3794.651198] LustreError: 245322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775835720 with bad export cookie 14951345879884463500 [ 3794.654280] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3794.656215] LustreError: 245322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3794.782840] Lustre: server umount lustre-MDT0001 complete [ 3798.488087] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 3799.713826] Key type lgssc unregistered [ 3799.848609] LNet: 249310:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3799.852267] LNetError: 249310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3799.861560] LNet: Removed LNI 192.168.204.148@tcp [ 3800.217840] Key type .llcrypt unregistered [ 3800.219535] Key type ._llcrypt unregistered [ 3807.373547] Key type ._llcrypt registered [ 3807.376033] Key type .llcrypt registered [ 3807.419376] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 3807.720532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3807.750595] alg: No test for adler32 (adler32-zlib) [ 3808.616932] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 3808.698112] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 3810.279148] Key type lgssc registered [ 3810.693635] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3813.271018] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3813.276831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3814.430252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3815.754683] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3818.650887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3818.776147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3820.039876] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3822.235068] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3823.518975] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3826.199664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3826.326716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3828.348949] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3830.372154] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3839.661507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3850.213384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3850.214722] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3850.215465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3850.223093] Lustre: Skipped 1 previous similar message [ 3850.227819] LustreError: Skipped 1 previous similar message [ 3851.352841] Lustre: server umount lustre-OST0000 complete [ 3854.816856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3854.822624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3854.832384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3854.836744] Lustre: Skipped 1 previous similar message [ 3860.448996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3860.452565] Lustre: Skipped 3 previous similar messages [ 3865.568082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3865.569983] Lustre: Skipped 1 previous similar message [ 3867.103210] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3867.185194] Lustre: server umount lustre-MDT0000 complete [ 3868.559372] LustreError: 250472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775835794 with bad export cookie 13835804426615724313 [ 3868.562589] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3868.565202] LustreError: 250472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3868.681267] Lustre: server umount lustre-MDT0001 complete [ 3872.615981] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 3873.764200] Key type lgssc unregistered [ 3873.920737] LNet: 253841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3873.926291] LNetError: 253841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3873.937583] LNet: Removed LNI 192.168.204.148@tcp [ 3874.328763] Key type .llcrypt unregistered [ 3874.330109] Key type ._llcrypt unregistered [ 3884.459662] Key type ._llcrypt registered [ 3884.461604] Key type .llcrypt registered [ 3884.507429] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 3884.907716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3884.933081] alg: No test for adler32 (adler32-zlib) [ 3885.783313] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 3885.866277] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 3887.455167] Key type lgssc registered [ 3887.795293] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3890.279582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3890.285252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3891.444443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3892.655108] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3895.383255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3895.500987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3896.784244] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3898.526790] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3899.731446] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3901.936077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3902.046519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3903.863898] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3905.542786] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3917.414386] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3932.128136] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3932.132143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3932.138708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3933.157775] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3933.157947] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3933.158101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3935.019487] Lustre: server umount lustre-OST0000 complete [ 3938.272476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3938.273237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3938.277085] Lustre: Skipped 1 previous similar message [ 3938.279977] Lustre: Skipped 2 previous similar messages [ 3943.392091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3943.394087] Lustre: Skipped 1 previous similar message [ 3948.512228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3948.514417] Lustre: Skipped 1 previous similar message [ 3951.071126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3951.161392] Lustre: server umount lustre-MDT0000 complete [ 3952.438140] LustreError: 255199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775835878 with bad export cookie 13970555472463514566 [ 3952.441705] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3952.442130] LustreError: 255199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3952.544983] Lustre: server umount lustre-MDT0001 complete [ 3956.228738] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 3957.349943] Key type lgssc unregistered [ 3957.501708] LNet: 259193:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3957.507363] LNetError: 259193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3957.518654] LNet: Removed LNI 192.168.204.148@tcp [ 3957.997962] Key type .llcrypt unregistered [ 3958.000346] Key type ._llcrypt unregistered [ 3966.221079] Key type ._llcrypt registered [ 3966.222835] Key type .llcrypt registered [ 3966.283358] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 3966.798453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3966.809790] alg: No test for adler32 (adler32-zlib) [ 3967.678453] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 3967.762836] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 3969.343177] Key type lgssc registered [ 3969.735930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3972.857950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3972.863940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3974.067235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3975.436768] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3979.095450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3979.215172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3980.459459] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3982.348846] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3983.797521] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3986.684614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3986.841891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3989.121473] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 3991.143635] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3999.142554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4009.442527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4009.442933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4009.448447] LustreError: Skipped 1 previous similar message [ 4009.449446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4009.454472] Lustre: Skipped 1 previous similar message [ 4012.927597] Lustre: server umount lustre-OST0000 complete [ 4015.073531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4015.078096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4015.085982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4015.088436] Lustre: Skipped 1 previous similar message [ 4019.682780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4019.682869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4019.704047] Lustre: Skipped 1 previous similar message [ 4021.077747] Lustre: server umount lustre-MDT0000 complete [ 4023.331839] LustreError: 260356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775835948 with bad export cookie 5217530555362260179 [ 4023.334539] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4023.337762] LustreError: 260356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4023.706738] Lustre: server umount lustre-MDT0001 complete [ 4029.074936] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 4030.189919] Key type lgssc unregistered [ 4030.320495] LNet: 263755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4030.322539] LNetError: 263755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4030.328430] LNet: Removed LNI 192.168.204.148@tcp [ 4030.609672] Key type .llcrypt unregistered [ 4030.610776] Key type ._llcrypt unregistered [ 4034.035545] Key type ._llcrypt registered [ 4034.037144] Key type .llcrypt registered [ 4034.079203] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4039.294175] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 4044.980040] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 4045.327083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4045.334448] alg: No test for adler32 (adler32-zlib) [ 4046.176148] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 4046.259328] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 4047.839150] Key type lgssc registered [ 4048.179330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4058.932755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4058.942668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4060.029715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4060.041456] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4060.082192] Lustre: lustre-MDT0000: new disk, initializing [ 4060.106995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4060.113715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4061.490474] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4065.779955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4065.824433] Lustre: 267224:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4065.850107] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4065.852015] Lustre: Skipped 1 previous similar message [ 4065.888513] Lustre: lustre-MDT0001: new disk, initializing [ 4065.921399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4065.937144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4065.941712] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4067.313676] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4070.378322] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4071.564276] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4073.663226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4073.768920] Lustre: lustre-OST0000: new disk, initializing [ 4073.770889] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4073.802510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4075.591959] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4078.525291] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4078.582491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4078.588066] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4078.619436] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4079.843410] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4079.913784] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4081.117445] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4081.183221] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4083.680509] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4083.682392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4083.683285] LustreError: Skipped 1 previous similar message [ 4083.683790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4083.687232] Lustre: Skipped 1 previous similar message [ 4083.693034] Lustre: Skipped 1 previous similar message [ 4087.885721] Lustre: server umount lustre-OST0000 complete [ 4091.871845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4091.875106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4091.879757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4093.920056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4093.920467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4095.187123] Lustre: server umount lustre-MDT0000 complete [ 4096.332895] LustreError: 267217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836021 with bad export cookie 17705714115357196182 [ 4096.335351] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4096.336043] LustreError: 267217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4096.436220] Lustre: server umount lustre-MDT0001 complete [ 4099.088360] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:47:03 (1775836023) [ 4101.074035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4101.224231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4102.354105] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4104.948710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4106.290162] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4108.031120] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4109.188559] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4111.282333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4111.330058] LustreError: 272818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4111.335261] LustreError: 272818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4111.373801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4111.376643] Lustre: Skipped 1 previous similar message [ 4113.235036] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4115.107748] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4119.521280] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4119.522106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4119.522830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4119.522834] Lustre: Skipped 1 previous similar message [ 4119.523578] LustreError: Skipped 1 previous similar message [ 4119.526938] Lustre: Skipped 1 previous similar message [ 4124.641088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4124.644624] Lustre: Skipped 2 previous similar messages [ 4124.944942] Lustre: server umount lustre-OST0000 complete [ 4129.761826] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4129.766448] Lustre: Skipped 1 previous similar message [ 4132.431421] Lustre: server umount lustre-MDT0000 complete [ 4133.635080] LustreError: 271210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836059 with bad export cookie 17705714115357197050 [ 4133.637632] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4133.639804] LustreError: 271210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4133.754976] Lustre: server umount lustre-MDT0001 complete [ 4137.281301] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 4138.249626] Key type lgssc unregistered [ 4138.381483] LNet: 274406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4138.383692] LNetError: 274406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4138.392419] LNet: Removed LNI 192.168.204.148@tcp [ 4138.659668] Key type .llcrypt unregistered [ 4138.660729] Key type ._llcrypt unregistered [ 4147.076586] Key type ._llcrypt registered [ 4147.078412] Key type .llcrypt registered [ 4147.119804] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 4153.206936] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 4153.720817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4153.735286] alg: No test for adler32 (adler32-zlib) [ 4154.597139] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 4154.683567] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 4156.271121] Key type lgssc registered [ 4156.656960] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4171.242451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4171.248593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4172.338281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4172.350193] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4172.386154] Lustre: lustre-MDT0000: new disk, initializing [ 4172.410500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4172.417581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4173.659318] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4177.826742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4177.866811] Lustre: 278170:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4177.891208] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4177.893168] Lustre: Skipped 1 previous similar message [ 4177.928899] Lustre: lustre-MDT0001: new disk, initializing [ 4177.961525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4177.975237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4177.979741] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4179.206534] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4182.230826] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4183.369250] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4185.560941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4185.657406] Lustre: lustre-OST2710: new disk, initializing [ 4185.659854] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4185.693954] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4187.635950] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4187.833294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4187.836725] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4187.875852] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4190.785924] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4193.152563] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4193.224073] Lustre: lustre-OST03e8: new disk, initializing [ 4193.226435] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4193.266367] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4193.550797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4193.553928] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4193.598256] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4195.216547] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4196.952164] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4199.473620] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4199.540331] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4200.813550] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4200.879787] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4202.165842] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4202.231346] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4203.468824] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4203.537739] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4208.608110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4208.612630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4208.620395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4209.121584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4209.122833] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4209.128844] Lustre: Skipped 2 previous similar messages [ 4210.648212] Lustre: server umount lustre-MDT0000 complete [ 4211.870342] LustreError: 278162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836137 with bad export cookie 5714423485343573517 [ 4211.873767] LustreError: 278162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4211.873776] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4211.991888] Lustre: server umount lustre-MDT0001 complete [ 4223.438332] Lustre: server umount lustre-OST2710 complete [ 4234.976359] Lustre: server umount lustre-OST03e8 complete [ 4240.990846] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 4244.112956] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 4256.253519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4256.341418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4256.352473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4256.397527] Lustre: lustre-MDT0000: new disk, initializing [ 4256.422113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4256.428827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4257.703538] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4262.065379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4262.103921] Lustre: 285883:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4262.203634] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4263.457088] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4266.627314] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4267.841740] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4270.083312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4270.192252] Lustre: lustre-OST0000: new disk, initializing [ 4270.193583] Lustre: Skipped 1 previous similar message [ 4270.195371] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4270.197546] Lustre: Skipped 2 previous similar messages [ 4270.228417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4270.230454] Lustre: Skipped 1 previous similar message [ 4271.989936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4271.993498] Lustre: Skipped 1 previous similar message [ 4271.995616] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4272.029773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4272.188689] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4275.227583] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4276.596312] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4276.665166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4278.074957] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4278.149697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4282.337211] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4282.339202] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4282.340625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4282.344045] Lustre: Skipped 1 previous similar message [ 4282.347394] Lustre: Skipped 4 previous similar messages [ 4285.071938] Lustre: server umount lustre-OST0000 complete [ 4287.456957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4287.457640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4287.461638] Lustre: Skipped 1 previous similar message [ 4287.465250] Lustre: Skipped 1 previous similar message [ 4292.586882] Lustre: server umount lustre-MDT0000 complete [ 4294.078211] LustreError: 285876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836219 with bad export cookie 5714423485343575183 [ 4294.081411] LustreError: 285876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4294.082054] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4294.195902] Lustre: server umount lustre-MDT0001 complete [ 4297.676877] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:50:22 (1775836222) [ 4298.250548] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4298.797585] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:50:23 (1775836223) [ 4299.814402] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 4309.396685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4309.503332] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4309.510868] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4309.576112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4310.929983] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4313.995512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4314.024214] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4314.033023] Lustre: 291309:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4315.486336] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4317.702681] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4319.176438] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4334.498192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4334.582121] Lustre: Denying initial registration attempt from nid 192.168.204.148@tcp, specified as failover [ 4334.584753] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4334.587655] LustreError: 292920:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -99 [ 4334.589906] LustreError: 292920:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 4334.591839] LustreError: 292920:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 4334.616266] Lustre: server umount lustre-OST0000 complete [ 4334.617583] LustreError: 292920:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4339.680727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4339.680946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4339.681335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4339.684048] Lustre: Skipped 1 previous similar message [ 4339.686201] LustreError: Skipped 1 previous similar message [ 4343.385726] Lustre: server umount lustre-MDT0000 complete [ 4344.576384] LustreError: 291300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836270 with bad export cookie 5714423485343576051 [ 4344.579124] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4344.580098] LustreError: 291300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4344.691471] Lustre: server umount lustre-MDT0001 complete [ 4348.176775] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 4349.231965] Key type lgssc unregistered [ 4349.359438] LNet: 294025:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4349.362085] LNetError: 294025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4349.368409] LNet: Removed LNI 192.168.204.148@tcp [ 4349.669508] Key type .llcrypt unregistered [ 4349.670546] Key type ._llcrypt unregistered [ 4353.144793] Key type ._llcrypt registered [ 4353.146486] Key type .llcrypt registered [ 4353.187342] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 4353.603644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4353.652172] alg: No test for adler32 (adler32-zlib) [ 4354.513602] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 4354.599050] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 4356.175154] Key type lgssc registered [ 4356.566165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4390.902894] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 4394.046029] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4394.051532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4395.140797] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4395.147843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4395.209359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4396.505585] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4399.674666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4399.729351] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4399.748569] Lustre: 296736:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4399.887059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4401.161765] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4403.076716] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4404.371901] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4419.617854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4419.693280] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4419.756441] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4421.565719] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4423.325668] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4424.080942] Lustre: server umount lustre-OST0000 complete [ 4425.696060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4425.698898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4425.703288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4428.783169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4428.784147] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4431.466431] Lustre: server umount lustre-MDT0000 complete [ 4432.730222] LustreError: 297459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836358 with bad export cookie 4917704311984587999 [ 4432.733905] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4432.734035] LustreError: 297459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4432.852457] Lustre: server umount lustre-MDT0001 complete [ 4436.529607] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 4437.539055] Key type lgssc unregistered [ 4437.668303] LNet: 299939:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4437.671145] LNetError: 299939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4437.679307] LNet: Removed LNI 192.168.204.148@tcp [ 4437.953948] Key type .llcrypt unregistered [ 4437.955490] Key type ._llcrypt unregistered [ 4446.817797] Key type ._llcrypt registered [ 4446.819400] Key type .llcrypt registered [ 4446.858090] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 4447.207329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4447.231386] alg: No test for adler32 (adler32-zlib) [ 4448.084223] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 4448.164517] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 4449.743191] Key type lgssc registered [ 4450.184255] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4452.960367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4452.966251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4454.112623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4455.641365] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4459.619086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4459.769778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4461.251249] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4463.638223] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4465.127500] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4469.538479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4469.689889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4471.680488] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4473.664794] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4484.067444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4484.069169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4484.073586] Lustre: Skipped 1 previous similar message [ 4486.882507] Lustre: server umount lustre-MDT0000 complete [ 4488.098289] LustreError: 301297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836413 with bad export cookie 990499800027020842 [ 4488.100827] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4488.104273] LustreError: 301297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4489.183861] LustreError: 301310:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4489.184063] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4489.189081] LustreError: 301310:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4489.193129] Lustre: Skipped 1 previous similar message [ 4489.198167] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4489.200257] Lustre: Skipped 3 previous similar messages [ 4493.216649] LustreError: 301312:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4493.216946] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4493.226921] LustreError: 301312:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4493.235292] Lustre: Skipped 1 previous similar message [ 4494.257134] Lustre: server umount lustre-MDT0001 complete [ 4496.424619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4499.252939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4512.736002] LustreError: 300626:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001b40a399 x1862099400677760/t0(0) o250->MGC192.168.204.148@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4512.898302] LustreError: 302918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4512.930643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4512.954797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4514.427137] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4517.670874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4517.838926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4518.885892] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4519.458969] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4521.507399] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4522.032337] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4522.983130] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4523.201771] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4529.864219] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 6 sec [ 4533.218632] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4533.221189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4533.221408] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4533.222086] Lustre: Skipped 1 previous similar message [ 4537.252984] Lustre: server umount lustre-OST0000 complete [ 4543.457113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4543.457238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4543.458328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4543.458333] Lustre: Skipped 1 previous similar message [ 4543.461171] Lustre: Skipped 1 previous similar message [ 4544.857441] Lustre: server umount lustre-MDT0000 complete [ 4546.119204] LustreError: 301296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836471 with bad export cookie 990499800027027996 [ 4546.121873] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4546.123096] LustreError: 301296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4546.236530] Lustre: server umount lustre-MDT0001 complete [ 4549.867492] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 4550.902832] Key type lgssc unregistered [ 4551.026305] LNet: 307003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4551.028933] LNetError: 307003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4551.035324] LNet: Removed LNI 192.168.204.148@tcp [ 4551.333514] Key type .llcrypt unregistered [ 4551.334600] Key type ._llcrypt unregistered [ 4559.834952] Key type ._llcrypt registered [ 4559.835952] Key type .llcrypt registered [ 4559.873985] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 4560.298302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4560.321944] alg: No test for adler32 (adler32-zlib) [ 4561.168754] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 4561.249278] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 4562.831144] Key type lgssc registered [ 4563.148512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4565.551788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4565.557376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4566.705764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4568.065801] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4570.923492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4571.043053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4572.335539] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4574.198317] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4575.375856] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4580.449322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4580.450433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4580.455448] Lustre: Skipped 1 previous similar message [ 4582.481583] Lustre: server umount lustre-MDT0000 complete [ 4583.664206] LustreError: 309769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836509 with bad export cookie 8951359016169194301 [ 4583.666900] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4583.670573] LustreError: 309769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4583.774288] Lustre: server umount lustre-MDT0001 complete [ 4588.380712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4588.475852] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4588.485904] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4588.558888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4589.879490] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4592.675139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4592.704153] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4592.713742] Lustre: 310739:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4592.811478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4594.064699] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4595.871147] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4597.074668] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4601.633972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4601.701842] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4601.705122] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4601.709393] LustreError: 312395:0:(tgt_mount.c:2545:server_fill_super()) Unable to start targets: -2 [ 4601.712285] LustreError: 312395:0:(tgt_mount.c:2063:server_put_super()) no obd lustre-OST0000 [ 4601.714276] LustreError: 312395:0:(tgt_mount.c:114:server_deregister_mount()) lustre-OST0000 not registered [ 4601.738183] Lustre: server umount lustre-OST0000 complete [ 4601.739829] LustreError: 312395:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4606.561974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4606.625924] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4606.686367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4608.501642] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4610.336982] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4612.880844] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4612.910637] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4612.934133] Lustre: lustre-OST0001: new disk, initializing [ 4612.935908] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4613.830914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4613.834644] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4613.857046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4613.878676] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4614.792233] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4616.777971] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4619.235615] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4619.236156] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4619.239875] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4619.242247] LustreError: Skipped 1 previous similar message [ 4619.247555] Lustre: Skipped 2 previous similar messages [ 4619.851289] Lustre: server umount lustre-OST0001 complete [ 4624.356642] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4624.358323] LustreError: 312862:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4624.358362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4624.358379] Lustre: Skipped 1 previous similar message [ 4624.359163] LustreError: Skipped 1 previous similar message [ 4624.360251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4624.370536] LustreError: 312862:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4627.347615] Lustre: server umount lustre-OST0000 complete [ 4628.960678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4628.966136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4628.974660] Lustre: Skipped 1 previous similar message [ 4628.978722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4628.982588] Lustre: Skipped 1 previous similar message [ 4634.592760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4634.599761] Lustre: Skipped 3 previous similar messages [ 4634.800044] Lustre: server umount lustre-MDT0000 complete [ 4636.125188] LustreError: 310729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836561 with bad export cookie 8951359016169194840 [ 4636.129343] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4636.129592] LustreError: 310729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4636.248629] Lustre: server umount lustre-MDT0001 complete [ 4639.996771] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 4641.012468] Key type lgssc unregistered [ 4641.138338] LNet: 315643:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4641.140929] LNetError: 315643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4641.148299] LNet: Removed LNI 192.168.204.148@tcp [ 4641.532540] Key type .llcrypt unregistered [ 4641.534540] Key type ._llcrypt unregistered [ 4671.920900] Key type ._llcrypt registered [ 4671.922132] Key type .llcrypt registered [ 4671.957466] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 4678.323683] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 4678.664752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4678.705864] alg: No test for adler32 (adler32-zlib) [ 4679.558380] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 4679.642065] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 4681.223132] Key type lgssc registered [ 4681.543896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4692.967548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4692.974064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4694.063509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4694.075838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4694.117124] Lustre: lustre-MDT0000: new disk, initializing [ 4694.142573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4694.149188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4695.436952] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4699.585147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4699.626333] Lustre: 320511:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4699.650707] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4699.652642] Lustre: Skipped 1 previous similar message [ 4699.692669] Lustre: lustre-MDT0001: new disk, initializing [ 4699.723533] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4699.735116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4699.738367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4701.027743] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4704.071195] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4705.330823] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4707.631357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4707.730832] Lustre: lustre-OST0000: new disk, initializing [ 4707.732871] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4707.758875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4709.736229] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4711.672980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4711.676809] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4711.705319] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4712.929527] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4714.246603] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4714.318597] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4715.600172] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4715.667565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4717.024514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4717.026281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4717.026914] LustreError: Skipped 1 previous similar message [ 4717.031963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4717.035010] Lustre: Skipped 1 previous similar message [ 4718.348723] Lustre: server umount lustre-OST0000 complete [ 4720.096005] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4720.099084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4720.103615] Lustre: Skipped 1 previous similar message [ 4720.105853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4722.144100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4722.144809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4725.722046] Lustre: server umount lustre-MDT0000 complete [ 4726.971997] LustreError: 320503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836652 with bad export cookie 13105137346831968325 [ 4726.974939] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4726.976043] LustreError: 320503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4727.077714] Lustre: server umount lustre-MDT0001 complete [ 4729.736091] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:57:34 (1775836654) [ 4732.084789] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:57:36 (1775836656) [ 4735.089878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4735.252883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4736.492884] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4739.172223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4739.293868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4740.572904] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4742.323176] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4743.483094] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4745.656593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4747.514674] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4749.272896] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4768.197467] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4768.754922] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4769.252293] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4769.790879] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4770.409758] Lustre: Failing over lustre-MDT0000 [ 4770.657325] Lustre: server umount lustre-MDT0000 complete [ 4774.989770] LustreError: 324848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4774.994572] LustreError: 324848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4775.392647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4775.393278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4775.400961] Lustre: Skipped 2 previous similar messages [ 4780.102662] LustreError: 324848:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.48@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4780.107819] LustreError: 324848:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4783.479651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4783.519291] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4783.615309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4783.617712] Lustre: Skipped 1 previous similar message [ 4783.640392] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4784.877096] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4785.222749] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4788.712871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4788.722824] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4788.740064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4790.071458] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4790.624795] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4792.544821] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4793.170185] Lustre: Failing over lustre-MDT0000 [ 4793.323800] Lustre: server umount lustre-MDT0000 complete [ 4793.824864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4793.826533] LustreError: 324847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4793.826908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4793.829116] Lustre: Skipped 1 previous similar message [ 4793.837887] LustreError: 324847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4798.944080] LustreError: 324852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4798.949075] LustreError: 324852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4804.064369] LustreError: 324871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4804.069803] LustreError: 324871:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4806.271051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4806.310674] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4806.402979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4806.424210] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4807.750193] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4809.811455] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4811.749121] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4811.751411] Lustre: Skipped 2 previous similar messages [ 4811.759215] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4811.776164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4812.916457] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4813.440848] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4816.865440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4816.866429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4816.866593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4816.866603] Lustre: Skipped 1 previous similar message [ 4816.868737] Lustre: Skipped 3 previous similar messages [ 4821.776716] Lustre: server umount lustre-MDT0000 complete [ 4821.984901] LustreError: 324871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4821.989368] LustreError: 324871:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4823.009217] LustreError: 324879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836748 with bad export cookie 13105137346831970978 [ 4823.013495] LustreError: 324879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4823.013806] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4823.143917] Lustre: server umount lustre-MDT0001 complete [ 4834.716430] Lustre: server umount lustre-OST0000 complete [ 4839.930162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4840.091471] LustreError: 330648:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4840.115754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4841.494194] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4844.275333] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4845.663953] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4847.361032] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4848.533896] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4850.801509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4852.851905] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4854.632473] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4865.326135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 4874.720294] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4874.724846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4874.732341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4874.736674] Lustre: Skipped 2 previous similar messages [ 4877.579518] Lustre: server umount lustre-OST0000 complete [ 4885.984247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4885.986251] Lustre: Skipped 5 previous similar messages [ 4893.151209] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4893.238356] Lustre: server umount lustre-MDT0000 complete [ 4894.721857] LustreError: 330629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836820 with bad export cookie 13105137346831971706 [ 4894.724969] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4894.726759] LustreError: 330629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4894.858600] Lustre: server umount lustre-MDT0001 complete [ 4898.735655] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 4899.921412] Key type lgssc unregistered [ 4900.062422] LNet: 333847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4900.067922] LNetError: 333847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4900.079501] LNet: Removed LNI 192.168.204.148@tcp [ 4900.456104] Key type .llcrypt unregistered [ 4900.457837] Key type ._llcrypt unregistered [ 4910.277469] Key type ._llcrypt registered [ 4910.279275] Key type .llcrypt registered [ 4910.322570] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 4910.643756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4910.650133] alg: No test for adler32 (adler32-zlib) [ 4911.508404] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 4911.593218] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 4913.175082] Key type lgssc registered [ 4913.523958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4916.306408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4916.315187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4917.488650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4918.910962] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4922.453268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4922.590426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4924.037745] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4925.948359] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4927.264945] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4929.805078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4929.924340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4931.963612] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4934.192149] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4943.275425] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 4953.573219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4953.574165] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4953.575463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4953.580735] Lustre: Skipped 2 previous similar messages [ 4954.946319] Lustre: server umount lustre-MDT0000 complete [ 4956.280361] LustreError: 336042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836881 with bad export cookie 8759386201808890701 [ 4956.286012] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4956.287853] LustreError: 336042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4956.446562] Lustre: server umount lustre-MDT0001 complete [ 4968.343448] Lustre: server umount lustre-OST0000 complete [ 4973.200035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4973.370651] LustreError: 338732:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4973.376750] LustreError: 338732:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4973.391362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4974.715397] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4978.053153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4978.159939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4979.752080] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4981.766368] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4982.931277] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4985.023336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4986.867543] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 4988.528663] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4998.444993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5008.869315] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5008.871445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5008.876580] Lustre: Skipped 2 previous similar messages [ 5008.876908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5008.882747] Lustre: Skipped 4 previous similar messages [ 5012.784934] Lustre: server umount lustre-MDT0000 complete [ 5013.988771] LustreError: 338751:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5013.999478] LustreError: 338751:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5014.538582] LustreError: 338714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775836940 with bad export cookie 8759386201808891800 [ 5014.545264] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5014.545420] LustreError: 338714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5014.704831] Lustre: server umount lustre-MDT0001 complete [ 5026.861475] Lustre: server umount lustre-OST0000 complete [ 5033.449823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5033.617557] LustreError: 342326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5033.648094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5033.652141] Lustre: Skipped 1 previous similar message [ 5035.283170] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5038.734250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5040.670453] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5043.162428] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5044.741365] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5047.385150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5049.324352] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5051.130808] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5058.790108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5064.161645] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5064.162285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5064.169089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5064.172549] Lustre: Skipped 1 previous similar message [ 5068.942043] Lustre: server umount lustre-OST0000 complete [ 5074.401731] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5074.402382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5074.405164] Lustre: Skipped 2 previous similar messages [ 5076.389377] Lustre: server umount lustre-MDT0000 complete [ 5077.635546] LustreError: 344490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775837003 with bad export cookie 8759386201808892808 [ 5077.640207] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5077.641338] LustreError: 344490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5077.766623] Lustre: server umount lustre-MDT0001 complete [ 5081.404542] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5082.396116] Key type lgssc unregistered [ 5082.538505] LNet: 345524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5082.543923] LNetError: 345524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5082.552565] LNet: Removed LNI 192.168.204.148@tcp [ 5082.859240] Key type .llcrypt unregistered [ 5082.860417] Key type ._llcrypt unregistered [ 5092.948229] Key type ._llcrypt registered [ 5092.949454] Key type .llcrypt registered [ 5092.995990] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5093.321053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5093.375098] alg: No test for adler32 (adler32-zlib) [ 5094.248391] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 5094.332721] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 5095.911114] Key type lgssc registered [ 5096.233810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5098.641241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5098.646594] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5098.649087] LustreError: 346924:0:(osd_handler.c:8532:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5098.657907] LustreError: 346924:0:(obd_config.c:843:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5098.660165] LustreError: 346924:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5098.662407] LustreError: 346924:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5098.664732] LustreError: 346924:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5101.572941] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5101.575668] LustreError: 347321:0:(osd_handler.c:8532:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5101.584224] LustreError: 347321:0:(obd_config.c:843:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5101.588851] LustreError: 347321:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5101.592521] LustreError: 347321:0:(tgt_mount.c:2513:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5101.594928] LustreError: 347321:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5109.294730] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5110.405167] Key type lgssc unregistered [ 5110.554447] LNet: 348306:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5110.559409] LNetError: 348306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5110.571546] LNet: Removed LNI 192.168.204.148@tcp [ 5110.978386] Key type .llcrypt unregistered [ 5110.979518] Key type ._llcrypt unregistered [ 5117.048047] Key type ._llcrypt registered [ 5117.049140] Key type .llcrypt registered [ 5117.086282] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 5124.127893] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5124.483489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5124.549706] alg: No test for adler32 (adler32-zlib) [ 5125.395924] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 5125.474229] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 5127.055108] Key type lgssc registered [ 5127.380887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5140.398672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5140.406120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5141.501777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5141.512965] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5141.555443] Lustre: lustre-MDT0000: new disk, initializing [ 5141.584487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5141.591853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5142.772086] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5146.823313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5146.864287] Lustre: 351402:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5146.891434] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5146.894425] Lustre: Skipped 1 previous similar message [ 5146.938205] Lustre: lustre-MDT0001: new disk, initializing [ 5146.977177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5146.990259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5146.995607] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5148.312139] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5151.442603] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5152.565539] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5154.780911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5154.887855] Lustre: lustre-OST0000: new disk, initializing [ 5154.889542] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5154.915266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5156.732603] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5159.855395] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5161.385632] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5164.025122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5164.028342] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5164.058077] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5164.520281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5165.896241] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5165.964849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5169.122710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5169.123387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5169.123626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5169.127022] Lustre: Skipped 1 previous similar message [ 5172.776710] Lustre: server umount lustre-OST0000 complete [ 5177.823742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5177.826496] LustreError: Skipped 1 previous similar message [ 5177.827880] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5177.832140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5177.834086] Lustre: Skipped 1 previous similar message [ 5179.362347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5179.363665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5180.529536] Lustre: server umount lustre-MDT0000 complete [ 5182.220867] LustreError: 351394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775837107 with bad export cookie 13939937027062287260 [ 5182.226760] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5182.229202] LustreError: 351394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5182.363640] Lustre: server umount lustre-MDT0001 complete [ 5185.134420] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5185.720464] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:05:10 (1775837110) [ 5188.057809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5188.257555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5189.760702] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5193.181258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5193.343118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5194.707455] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5196.474643] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5197.862662] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5200.844163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5203.319232] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5205.362010] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5207.537252] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5207.600562] Lustre: lustre-OST0001: new disk, initializing [ 5207.602504] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5207.631787] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5207.633815] Lustre: Skipped 1 previous similar message [ 5209.405204] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5212.506821] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5214.206134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5214.209309] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5214.240431] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5218.894068] Lustre: lustre-OST0001: Not available for connect from 192.168.204.48@tcp (stopping) [ 5218.896173] Lustre: Skipped 1 previous similar message [ 5219.300721] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5219.302935] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5220.476495] Lustre: server umount lustre-OST0001 complete [ 5224.422681] LustreError: 357067:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5224.427503] LustreError: 357067:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5229.542549] LustreError: 357068:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5229.547332] LustreError: 357068:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5233.632074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5233.635461] LustreError: Skipped 1 previous similar message [ 5233.636963] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5233.640893] Lustre: Skipped 1 previous similar message [ 5233.642719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5233.644666] Lustre: Skipped 2 previous similar messages [ 5234.658639] LustreError: 357067:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5234.663603] LustreError: 357067:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5238.538870] Lustre: server umount lustre-OST0000 complete [ 5239.775663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5239.777989] LustreError: Skipped 1 previous similar message [ 5244.896529] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5244.899683] Lustre: Skipped 5 previous similar messages [ 5245.972510] Lustre: server umount lustre-MDT0000 complete [ 5247.124069] LustreError: 355456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775837172 with bad export cookie 13939937027062288121 [ 5247.127240] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5247.127607] LustreError: 355456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5253.291518] Lustre: server umount lustre-MDT0001 complete [ 5256.815820] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5257.795858] Key type lgssc unregistered [ 5257.917511] LNet: 359826:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5257.919708] LNetError: 359826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5257.928437] LNet: Removed LNI 192.168.204.148@tcp [ 5258.222881] Key type .llcrypt unregistered [ 5258.224130] Key type ._llcrypt unregistered [ 5275.326119] Key type ._llcrypt registered [ 5275.327767] Key type .llcrypt registered [ 5275.361320] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5275.760873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5275.773506] alg: No test for adler32 (adler32-zlib) [ 5276.625057] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 5276.702370] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 5278.279101] Key type lgssc registered [ 5278.603748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5280.970321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5280.975882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5282.057093] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5282.064446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5282.120569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5283.427428] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5286.140040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5286.167666] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5286.177052] Lustre: 362584:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5286.262514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5287.436275] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5289.126451] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5290.237672] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5291.488811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5291.488912] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5291.489545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5291.492479] Lustre: Skipped 1 previous similar message [ 5296.608477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5296.611068] Lustre: Skipped 2 previous similar messages [ 5301.729627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5301.734413] Lustre: Skipped 1 previous similar message [ 5305.311121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5305.369571] Lustre: server umount lustre-MDT0000 complete [ 5307.058186] LustreError: 363306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775837232 with bad export cookie 13852402211328912695 [ 5307.060471] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5307.061606] LustreError: 363306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5307.163104] Lustre: server umount lustre-MDT0001 complete [ 5310.400105] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:07:15 (1775837235) [ 5312.820201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5312.961617] LustreError: 364859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5312.968913] LustreError: 364859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5312.977641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5314.260206] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5317.483983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5317.604549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5318.862507] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5320.714410] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5321.852866] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5324.055178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5324.139060] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5324.209431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5326.176550] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5327.985823] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5331.632872] Lustre: Permanently deactivating lustre-OST0000 [ 5331.637699] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5335.162051] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5336.256057] LustreError: 367452:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5336.260014] LustreError: 367452:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5336.262955] LustreError: 367452:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5336.894407] LustreError: 367500:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5336.897482] LustreError: 367500:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5343.630860] Lustre: server umount lustre-OST0000 complete [ 5344.500778] LustreError: 367750:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [ 5344.503193] LustreError: 367750:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5348.320823] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5348.320905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5348.324207] Lustre: Skipped 1 previous similar message [ 5348.324482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5353.440056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5353.442295] Lustre: Skipped 2 previous similar messages [ 5354.489644] Lustre: server umount lustre-MDT0000 complete [ 5355.780530] LustreError: 364840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775837281 with bad export cookie 13852402211328913192 [ 5355.785100] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5355.785756] LustreError: 364840:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5366.297766] Lustre: server umount lustre-MDT0001 complete [ 5369.323031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5370.652387] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5371.591595] LustreError: 368889:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5373.607006] Lustre: 369228:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.148@tcp [ 5374.273380] Lustre: server umount MGS complete [ 5378.036290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5378.192370] LustreError: 369951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5378.197072] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5378.200730] Lustre: Skipped 1 previous similar message [ 5378.222923] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5379.519617] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5383.068248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5383.163798] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5384.734475] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5386.983879] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5388.455953] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5389.118946] Lustre: Permanently reactivating lustre-OST0000 [ 5389.124563] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5389.126411] Lustre: Skipped 2 previous similar messages [ 5411.807084] LustreError: 371409:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5415.535332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5415.698123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5415.702562] Lustre: Skipped 1 previous similar message [ 5417.862991] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5420.007966] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5432.289737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5432.290085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5432.290690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5432.300938] Lustre: Skipped 1 previous similar message [ 5438.244838] Lustre: server umount lustre-OST0000 complete [ 5442.528659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5445.846956] Lustre: server umount lustre-MDT0000 complete [ 5446.955439] LustreError: 369933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775837372 with bad export cookie 13852402211328914228 [ 5446.958498] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5446.959217] LustreError: 369933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5447.076389] Lustre: server umount lustre-MDT0001 complete [ 5450.454246] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5451.486557] Key type lgssc unregistered [ 5451.613368] LNet: 374174:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5451.615633] LNetError: 374174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5451.621457] LNet: Removed LNI 192.168.204.148@tcp [ 5451.913096] Key type .llcrypt unregistered [ 5451.914234] Key type ._llcrypt unregistered [ 5457.401418] Key type ._llcrypt registered [ 5457.402741] Key type .llcrypt registered [ 5457.439864] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 5463.397326] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5463.699532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5463.738140] alg: No test for adler32 (adler32-zlib) [ 5464.575220] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 5464.647527] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 5466.223118] Key type lgssc registered [ 5466.558772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5476.679044] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5476.684243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5477.765720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5477.777428] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5477.818433] Lustre: lustre-MDT0000: new disk, initializing [ 5477.840679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5477.847281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5478.966086] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5482.757840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5482.795060] Lustre: 377270:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5482.816281] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5482.818073] Lustre: Skipped 1 previous similar message [ 5482.853291] Lustre: lustre-MDT0001: new disk, initializing [ 5482.882836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5482.892978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5482.895755] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5483.996696] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5486.894361] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5488.005869] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5489.999988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5490.090567] Lustre: lustre-OST0000: new disk, initializing [ 5490.092434] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5490.117069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5491.693669] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5494.641683] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5495.799241] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5495.802544] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5495.840392] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5495.925457] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5495.995442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5497.188662] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5497.253421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5500.902685] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5500.902881] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5500.908289] LustreError: Skipped 1 previous similar message [ 5500.918357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5500.924825] Lustre: Skipped 1 previous similar message [ 5503.983682] Lustre: server umount lustre-OST0000 complete [ 5506.016564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5506.017299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5506.020045] Lustre: Skipped 2 previous similar messages [ 5506.023081] Lustre: Skipped 1 previous similar message [ 5511.136150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5511.137978] Lustre: Skipped 1 previous similar message [ 5511.644321] Lustre: server umount lustre-MDT0000 complete [ 5512.810061] LustreError: 378046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775837438 with bad export cookie 9345078127141801924 [ 5512.813439] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5512.813703] LustreError: 378046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5512.922696] Lustre: server umount lustre-MDT0001 complete [ 5515.486192] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5516.013449] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:10:40 (1775837440) [ 5518.044760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5518.201821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5519.306974] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5521.960853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5523.296149] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5524.988462] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5526.098307] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5528.269069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5528.388495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5528.391588] Lustre: Skipped 1 previous similar message [ 5530.205130] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5531.900972] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5532.486157] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5533.913205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5536.737429] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5536.738309] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5536.740094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5536.742812] Lustre: Skipped 1 previous similar message [ 5541.131947] Lustre: server umount lustre-OST0000 complete [ 5542.879951] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5542.883163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5542.887634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5542.891142] Lustre: Skipped 1 previous similar message [ 5546.977685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5548.447543] Lustre: server umount lustre-MDT0000 complete [ 5549.634475] LustreError: 381300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775837475 with bad export cookie 9345078127141802792 [ 5549.638795] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5549.640446] LustreError: 381300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5549.769812] Lustre: server umount lustre-MDT0001 complete [ 5552.953505] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5553.944562] Key type lgssc unregistered [ 5554.068450] LNet: 384689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5554.070852] LNetError: 384689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5554.080520] LNet: Removed LNI 192.168.204.148@tcp [ 5554.413256] Key type .llcrypt unregistered [ 5554.414374] Key type ._llcrypt unregistered [ 5562.143752] Key type ._llcrypt registered [ 5562.145658] Key type .llcrypt registered [ 5562.191599] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5562.305276] Key type .llcrypt unregistered [ 5562.307227] Key type ._llcrypt unregistered [ 5569.361969] Key type ._llcrypt registered [ 5569.362952] Key type .llcrypt registered [ 5569.403414] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5569.789229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5569.809687] alg: No test for adler32 (adler32-zlib) [ 5570.676296] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 5570.761149] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 5572.343096] Key type lgssc registered [ 5572.747810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5575.357502] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5575.365305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5576.528029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5577.943242] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5580.866903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5581.000460] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5582.951393] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5584.727484] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5586.401212] LustreError: 386840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5586.406115] LustreError: 386840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5587.006112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5587.140586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5588.199935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5588.502607] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5592.544174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5592.547162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5592.550932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5593.569958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5593.570255] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5593.572646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5597.533184] Lustre: server umount lustre-OST0000 complete [ 5601.761522] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5601.765539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5601.772231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5601.774818] Lustre: Skipped 1 previous similar message [ 5603.808722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5603.809655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5605.009936] Lustre: server umount lustre-MDT0000 complete [ 5606.222380] LustreError: 386825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775837531 with bad export cookie 10969118748353582748 [ 5606.226152] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5606.227899] LustreError: 386825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5606.360268] Lustre: server umount lustre-MDT0001 complete [ 5610.113711] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5611.123427] Key type lgssc unregistered [ 5611.244336] LNet: 389832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5611.246369] LNetError: 389832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5611.256252] LNet: Removed LNI 192.168.204.148@tcp [ 5611.531304] Key type .llcrypt unregistered [ 5611.532680] Key type ._llcrypt unregistered [ 5620.144435] Key type ._llcrypt registered [ 5620.145440] Key type .llcrypt registered [ 5620.182933] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5620.456027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5620.485956] alg: No test for adler32 (adler32-zlib) [ 5621.335542] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 5621.416104] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 5622.991151] Key type lgssc registered [ 5623.338967] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5625.848499] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5625.854214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5641.311247] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5646.431183] LustreError: 391181:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.148@tcp: failed processing log, type 4: rc = -110 [ 5673.055158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5674.785100] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5676.420751] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5678.580217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5678.762949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5678.788601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5680.097480] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5682.858731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5682.990832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5684.266240] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5685.983778] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5687.219961] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5693.409923] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5693.411687] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5693.414973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5693.419604] Lustre: Skipped 1 previous similar message [ 5693.424222] Lustre: Skipped 1 previous similar message [ 5695.632753] Lustre: server umount lustre-OST0000 complete [ 5698.528947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5698.529147] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5698.529628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5698.540129] Lustre: Skipped 1 previous similar message [ 5703.004870] Lustre: server umount lustre-MDT0000 complete [ 5703.648896] LustreError: 392106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5703.654070] LustreError: 392106:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5704.212077] LustreError: 391189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775837629 with bad export cookie 12891307522180942395 [ 5704.215165] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5704.217917] LustreError: 391189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5704.325193] Lustre: server umount lustre-MDT0001 complete [ 5707.718439] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5708.658483] Key type lgssc unregistered [ 5708.777332] LNet: 394392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5708.780244] LNetError: 394392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5708.787348] LNet: Removed LNI 192.168.204.148@tcp [ 5709.074935] Key type .llcrypt unregistered [ 5709.076459] Key type ._llcrypt unregistered [ 5717.908178] Key type ._llcrypt registered [ 5717.909869] Key type .llcrypt registered [ 5717.948254] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5718.261207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5718.277210] alg: No test for adler32 (adler32-zlib) [ 5719.135446] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 5719.219609] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 5720.799080] Key type lgssc registered [ 5721.174791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5723.731319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5723.737866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5724.886176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5726.155337] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5729.020874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5729.165336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5730.593941] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5732.444874] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5733.650930] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5735.897366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5736.014506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5737.839750] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5739.593574] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5750.374897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5759.968654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5759.972325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5759.979054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5760.589368] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.48@tcp (stopping) [ 5760.993803] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5760.998886] Lustre: Skipped 1 previous similar message [ 5762.518134] Lustre: server umount lustre-MDT0000 complete [ 5763.256757] LustreError: 395749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.48@tcp arrived at 1775837688 with bad export cookie 13877697616014027968 [ 5763.262440] LustreError: 395749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5763.905464] LustreError: 395748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775837689 with bad export cookie 13877697616014027317 [ 5763.909953] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5765.087853] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5765.091201] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5765.095140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5765.096736] Lustre: Skipped 3 previous similar messages [ 5766.112000] LustreError: 395768:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5766.117808] LustreError: 395768:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5768.159948] LustreError: 395769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5768.160247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5768.167154] LustreError: 395769:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5768.169097] Lustre: Skipped 1 previous similar message [ 5769.997313] Lustre: server umount lustre-OST0000 complete [ 5772.297646] Lustre: server umount lustre-MDT0001 complete [ 5775.856021] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5776.907980] Key type lgssc unregistered [ 5777.038388] LNet: 399118:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5777.041163] LNetError: 399118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5777.048279] LNet: Removed LNI 192.168.204.148@tcp [ 5777.325604] Key type .llcrypt unregistered [ 5777.327129] Key type ._llcrypt unregistered [ 5786.109611] Key type ._llcrypt registered [ 5786.111321] Key type .llcrypt registered [ 5786.149472] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5786.491367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5786.532597] alg: No test for adler32 (adler32-zlib) [ 5787.391820] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 5787.476869] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 5789.055083] Key type lgssc registered [ 5789.423719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5791.988826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5791.994417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5793.146853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5794.292785] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5796.907607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5797.028447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5798.245620] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5799.933715] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5801.127563] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5803.301619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5803.430641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5804.453597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5805.255560] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5807.092732] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5811.681905] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5811.682649] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5811.685763] Lustre: Skipped 1 previous similar message [ 5811.689564] Lustre: Skipped 1 previous similar message [ 5813.326249] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.48@tcp (stopping) [ 5815.060836] Lustre: server umount lustre-MDT0001 complete [ 5816.800202] LustreError: 400493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5816.801133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5816.804981] LustreError: 400493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5816.809819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5821.920255] LustreError: 401215:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5821.921660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5822.804315] Lustre: server umount lustre-OST0000 complete [ 5824.180819] Lustre: server umount lustre-MDT0000 complete [ 5828.790161] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5829.906586] Key type lgssc unregistered [ 5830.060406] LNet: 403822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5830.064532] LNetError: 403822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5830.072467] LNet: Removed LNI 192.168.204.148@tcp [ 5830.401321] Key type .llcrypt unregistered [ 5830.403087] Key type ._llcrypt unregistered [ 5838.313206] Key type ._llcrypt registered [ 5838.314488] Key type .llcrypt registered [ 5838.348503] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 5844.353281] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5844.807626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5844.816081] alg: No test for adler32 (adler32-zlib) [ 5845.662148] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 5845.746910] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 5847.327109] Key type lgssc registered [ 5847.687728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5859.604947] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5864.802883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5864.807720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5865.900332] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5865.911426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5865.950768] Lustre: lustre-MDT0000: new disk, initializing [ 5865.974184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5865.981557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5867.283627] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5873.223825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5873.320175] Lustre: lustre-OST0000: new disk, initializing [ 5873.322576] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5873.324712] Lustre: Skipped 1 previous similar message [ 5873.348755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5875.455888] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5878.599597] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5879.301877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5879.305642] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5879.314973] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5884.855907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5884.892148] Lustre: 407757:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5884.915160] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5884.950467] Lustre: lustre-MDT0001: new disk, initializing [ 5884.983332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5884.993731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 5884.996482] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 5886.147531] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5889.423017] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5889.496723] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5890.711377] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5892.814049] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 2 sec [ 5897.697307] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5897.697436] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5897.702728] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5897.708103] Lustre: Skipped 1 previous similar message [ 5898.318345] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.48@tcp (stopping) [ 5900.496900] Lustre: server umount lustre-MDT0001 complete [ 5907.938297] LustreError: 407767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5907.942930] LustreError: 407767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5913.056650] LustreError: 409972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5913.056967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5913.066465] Lustre: Skipped 1 previous similar message [ 5913.068762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5918.176687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5918.177264] LustreError: 407767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5923.296309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5923.296766] LustreError: 409972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5927.391114] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5927.446389] Lustre: server umount lustre-OST0000 complete [ 5928.416947] LustreError: 407767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5934.839890] Lustre: server umount lustre-MDT0000 complete [ 5939.272910] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5940.253756] Key type lgssc unregistered [ 5940.386365] LNet: 412343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5940.389992] LNetError: 412343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5940.404301] LNet: Removed LNI 192.168.204.148@tcp [ 5940.701662] Key type .llcrypt unregistered [ 5940.703262] Key type ._llcrypt unregistered [ 5953.590625] Key type ._llcrypt registered [ 5953.591825] Key type .llcrypt registered [ 5953.630400] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 5953.729283] Key type .llcrypt unregistered [ 5953.731054] Key type ._llcrypt unregistered [ 5960.317107] Key type ._llcrypt registered [ 5960.318335] Key type .llcrypt registered [ 5960.356858] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 5960.771630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5960.802961] alg: No test for adler32 (adler32-zlib) [ 5961.657804] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 5961.745388] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 5963.327134] Key type lgssc registered [ 5963.690884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5972.512764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5972.519522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5973.613216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5973.626980] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5973.668173] Lustre: lustre-MDT0000: new disk, initializing [ 5973.694631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5973.701789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5975.042449] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5979.264366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5979.307124] Lustre: 415655:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5979.330966] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5979.333915] Lustre: Skipped 1 previous similar message [ 5979.378782] Lustre: lustre-MDT0001: new disk, initializing [ 5979.414159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5979.426722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5979.429823] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5980.695398] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5983.818797] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5985.072518] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5987.405473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5987.535449] Lustre: lustre-OST0000: new disk, initializing [ 5987.537799] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5987.571524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5988.064889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5988.069024] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5988.105079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5989.527258] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 5992.718514] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5998.561595] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5998.563247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5998.565773] Lustre: Skipped 2 previous similar messages [ 5998.568817] Lustre: Skipped 1 previous similar message [ 6001.248730] Lustre: server umount lustre-MDT0000 complete [ 6002.509258] LustreError: 415649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775837928 with bad export cookie 9149152067971578432 [ 6002.512078] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6002.513266] LustreError: 415649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6002.633832] Lustre: server umount lustre-MDT0001 complete [ 6014.105676] Lustre: server umount lustre-OST0000 complete [ 6017.498710] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:19:02 (1775837942) [ 6020.036820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6020.207729] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6021.567477] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6024.497378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6024.624527] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6025.941418] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6027.754671] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6028.998553] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6044.257979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6044.342393] Lustre: Found index 0 for lustre-OST0000, updating log [ 6044.403619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6045.416064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6046.331621] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6048.123810] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6050.338580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6050.339034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6050.346686] Lustre: Skipped 1 previous similar message [ 6051.404404] Lustre: server umount lustre-OST0000 complete [ 6065.631764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6065.634132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6065.637799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6067.680623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6067.681473] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6070.100559] Lustre: server umount lustre-MDT0000 complete [ 6071.267186] LustreError: 420924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775837996 with bad export cookie 9149152067971579811 [ 6071.269935] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6071.271035] LustreError: 420924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6071.378627] Lustre: server umount lustre-MDT0001 complete [ 6074.206606] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:19:59 (1775837999) [ 6110.239347] LNet: 1 local NIs in recovery (showing 1): 192.168.204.148@tcp [ 6676.987043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6677.278983] LustreError: 425379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6677.303506] LustreError: 425379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6677.363246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6681.050863] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6682.594637] LustreError: 425380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6686.690841] LustreError: 425379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6687.527069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6687.757553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6690.559927] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6694.512353] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6697.178207] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6714.856559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6714.993328] Lustre: Found index 0 for lustre-OST0000, updating log [ 6715.173224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6719.297557] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 6723.287407] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6725.793690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6725.801741] Lustre: Skipped 1 previous similar message [ 6725.805742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6730.981218] Lustre: server umount lustre-OST0000 complete [ 6733.795580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6733.806301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6733.819416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6733.822943] Lustre: Skipped 1 previous similar message [ 6734.880510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6734.883500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6736.993941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6736.997457] Lustre: Skipped 4 previous similar messages [ 6737.951373] LNet: 1 local NIs in recovery (showing 1): 192.168.204.148@tcp [ 6739.940916] Lustre: server umount lustre-MDT0000 complete [ 6740.064912] LustreError: 427643:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6740.083986] LustreError: 427643:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6742.682057] LustreError: 425360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775838668 with bad export cookie 9149152067971580868 [ 6742.682965] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6742.699967] LustreError: 425360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6742.900527] Lustre: server umount lustre-MDT0001 complete [ 6766.817928] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 6774.280609] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 6801.440243] LNet: 1 local NIs in recovery (showing 1): 192.168.204.148@tcp [ 6802.737908] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:32:06 (1775838726) [ 6816.622909] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 6818.472984] Key type lgssc unregistered [ 6818.746614] LNet: 431603:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6818.753635] LNetError: 431603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6939.637822] LNet: Removed LNI 192.168.204.148@tcp [ 6940.247180] Key type .llcrypt unregistered [ 6940.249398] Key type ._llcrypt unregistered [ 6962.953169] Key type ._llcrypt registered [ 6962.956626] Key type .llcrypt registered [ 6963.116942] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 6964.532757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6964.738807] alg: No test for adler32 (adler32-zlib) [ 6965.901744] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 6966.436521] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 6968.145096] Key type lgssc registered [ 6969.511148] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6976.365932] LNet: Added LNI 192.168.204.148@tcp100 [8/256/0/180] [ 6977.852704] LNet: Added LNI 192.168.204.148@tcp101 [8/256/0/180] [ 6979.201789] LNet: Added LNI 192.168.204.148@tcp102 [8/256/0/180] [ 6980.724722] LNet: Added LNI 192.168.204.148@tcp103 [8/256/0/180] [ 6982.281067] LNet: Added LNI 192.168.204.148@tcp104 [8/256/0/180] [ 6983.850756] LNet: Added LNI 192.168.204.148@tcp105 [8/256/0/180] [ 6985.524458] LNet: Added LNI 192.168.204.148@tcp106 [8/256/0/180] [ 6987.035267] LNet: Added LNI 192.168.204.148@tcp107 [8/256/0/180] [ 6988.675280] LNet: Added LNI 192.168.204.148@tcp108 [8/256/0/180] [ 6990.176041] LNet: Added LNI 192.168.204.148@tcp109 [8/256/0/180] [ 6991.321624] LNet: Added LNI 192.168.204.148@tcp110 [8/256/0/180] [ 6992.561982] LNet: Added LNI 192.168.204.148@tcp111 [8/256/0/180] [ 6994.233501] LNet: Added LNI 192.168.204.148@tcp112 [8/256/0/180] [ 6995.567407] LNet: Added LNI 192.168.204.148@tcp113 [8/256/0/180] [ 6996.784896] LNet: Added LNI 192.168.204.148@tcp114 [8/256/0/180] [ 6998.244276] LNet: Added LNI 192.168.204.148@tcp115 [8/256/0/180] [ 7000.000084] LNet: Added LNI 192.168.204.148@tcp116 [8/256/0/180] [ 7001.207951] LNet: Added LNI 192.168.204.148@tcp117 [8/256/0/180] [ 7002.290528] LNet: Added LNI 192.168.204.148@tcp118 [8/256/0/180] [ 7003.797712] LNet: Added LNI 192.168.204.148@tcp119 [8/256/0/180] [ 7004.966950] LNet: Added LNI 192.168.204.148@tcp120 [8/256/0/180] [ 7006.444560] LNet: Added LNI 192.168.204.148@tcp121 [8/256/0/180] [ 7007.803648] LNet: Added LNI 192.168.204.148@tcp122 [8/256/0/180] [ 7009.299129] LNet: Added LNI 192.168.204.148@tcp123 [8/256/0/180] [ 7010.896035] LNet: Added LNI 192.168.204.148@tcp124 [8/256/0/180] [ 7012.362965] LNet: Added LNI 192.168.204.148@tcp125 [8/256/0/180] [ 7013.614479] LNet: Added LNI 192.168.204.148@tcp126 [8/256/0/180] [ 7014.514707] LNet: Added LNI 192.168.204.148@tcp127 [8/256/0/180] [ 7015.391660] LNet: Added LNI 192.168.204.148@tcp128 [8/256/0/180] [ 7016.455697] LNet: Added LNI 192.168.204.148@tcp129 [8/256/0/180] [ 7017.434040] LNet: Added LNI 192.168.204.148@tcp130 [8/256/0/180] [ 7018.586464] LNet: Added LNI 192.168.204.148@tcp131 [8/256/0/180] [ 7019.594040] LNet: Added LNI 192.168.204.148@tcp132 [8/256/0/180] [ 7020.884557] LNet: Added LNI 192.168.204.148@tcp133 [8/256/0/180] [ 7021.986225] LNet: Added LNI 192.168.204.148@tcp134 [8/256/0/180] [ 7023.141124] LNet: Added LNI 192.168.204.148@tcp135 [8/256/0/180] [ 7031.687793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7031.735687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7032.896580] Lustre: 434622:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.148@tcp: can't add NID 192.168.204.148@tcp130: rc = -75 [ 7033.100559] LNetError: 432081:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.148 [ 7033.121562] LNetError: Protocol error connecting to 192.168.204.148@tcp118 at host 192.168.204.148:988: is it running a compatible version of Lustre? [ 7033.139527] LNetError: Skipped 3 previous similar messages [ 7033.210165] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7033.271673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7033.396796] Lustre: lustre-MDT0000: new disk, initializing [ 7033.536854] Lustre: 434677:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.148@tcp: can't add NID 192.168.204.148@tcp130: rc = -75 [ 7033.546403] Lustre: 434677:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 7033.586145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7033.614875] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7034.016207] LNet: 9 local NIs in recovery (showing 9): 192.168.204.148@tcp122, 192.168.204.148@tcp123, 192.168.204.148@tcp124, 192.168.204.148@tcp118, 192.168.204.148@tcp125, 192.168.204.148@tcp126, 192.168.204.148@tcp128, 192.168.204.148@tcp129, 192.168.204.148@tcp127 [ 7034.016636] LNetError: 432080:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.148 [ 7034.065892] LNetError: 432080:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 10 previous similar messages [ 7034.087336] LNetError: Protocol error connecting to 192.168.204.148@tcp111 at host 192.168.204.148:988: is it running a compatible version of Lustre? [ 7034.100893] LNetError: Skipped 11 previous similar messages [ 7036.128695] LNetError: 434646:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.148 [ 7036.148969] LNetError: 434646:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 5 previous similar messages [ 7036.173316] LNetError: Protocol error connecting to 192.168.204.148@tcp127 at host 192.168.204.148:988: is it running a compatible version of Lustre? [ 7038.553096] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7040.227366] LNetError: 432079:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.148 [ 7040.250636] LNetError: Protocol error connecting to 192.168.204.148@tcp127 at host 192.168.204.148:988: is it running a compatible version of Lustre? [ 7048.418247] LNetError: 432081:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.148 [ 7048.445645] LNetError: Protocol error connecting to 192.168.204.148@tcp127 at host 192.168.204.148:988: is it running a compatible version of Lustre? [ 7049.924171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7050.074844] Lustre: 434637:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7050.129849] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7050.135940] Lustre: Skipped 1 previous similar message [ 7050.200545] Lustre: lustre-MDT0001: new disk, initializing [ 7050.234118] Lustre: 435423:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.148@tcp: can't add NID 192.168.204.148@tcp130: rc = -75 [ 7050.242961] Lustre: 435423:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 7050.340395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7050.376468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7050.391473] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7055.669642] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7057.960101] Lustre: 435757:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.148@tcp: can't add NID 192.168.204.148@tcp130: rc = -75 [ 7057.971864] Lustre: 435757:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7063.841154] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7064.800811] LNetError: 434655:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.148 [ 7064.810664] LNetError: Protocol error connecting to 192.168.204.148@tcp127 at host 192.168.204.148:988: is it running a compatible version of Lustre? [ 7067.576801] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7074.432935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7074.580809] Lustre: lustre-OST0000: new disk, initializing [ 7074.586440] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7074.637618] Lustre: 436396:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.148@tcp: can't add NID 192.168.204.148@tcp130: rc = -75 [ 7074.642349] Lustre: 436396:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 7074.669314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7078.532563] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7082.060923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7082.066961] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7082.165841] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7082.272261] LNetError: 434642:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.148 [ 7082.277630] LNetError: 434642:0:(socklnd_cb.c:1815:ksocknal_recv_hello()) Skipped 9 previous similar messages [ 7082.282866] LNetError: Protocol error connecting to 192.168.204.148@tcp121 at host 192.168.204.148:988: is it running a compatible version of Lustre? [ 7082.290441] LNetError: Skipped 9 previous similar messages [ 7084.113714] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7092.709182] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7092.714445] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7092.721281] Lustre: Skipped 1 previous similar message [ 7092.722400] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7092.726506] Lustre: Skipped 1 previous similar message [ 7093.471260] LNet: 1 local NIs in recovery (showing 1): 192.168.204.148@tcp127 [ 7095.129804] Lustre: server umount lustre-OST0000 complete [ 7097.828439] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7097.847230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7097.852928] Lustre: Skipped 1 previous similar message [ 7102.948947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7102.958480] Lustre: Skipped 1 previous similar message [ 7103.731970] Lustre: server umount lustre-MDT0000 complete [ 7106.543132] LustreError: 436390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775839032 with bad export cookie 16037137822364316993 [ 7106.550333] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7106.559280] LustreError: 436390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7106.805485] Lustre: server umount lustre-MDT0001 complete [ 7114.564123] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 7116.661247] Key type lgssc unregistered [ 7116.955531] LNet: 438175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7116.961788] LNetError: 438175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7117.983581] LNet: Removed LNI 192.168.204.148@tcp [ 7117.989669] LNetError: 438175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7117.999686] LNet: Removed LNI 192.168.204.148@tcp100 [ 7119.007134] LNet: Removed LNI 192.168.204.148@tcp101 [ 7119.014772] LNetError: 438175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7119.024774] LNetError: 438175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7119.033471] LNet: Removed LNI 192.168.204.148@tcp102 [ 7120.095696] LNet: Removed LNI 192.168.204.148@tcp103 [ 7121.119362] LNet: Removed LNI 192.168.204.148@tcp104 [ 7121.129085] LNetError: 438175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7121.140524] LNetError: 438175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7121.146193] LNet: Removed LNI 192.168.204.148@tcp105 [ 7122.207210] LNet: Removed LNI 192.168.204.148@tcp106 [ 7123.232948] LNet: Removed LNI 192.168.204.148@tcp107 [ 7124.255189] LNet: Removed LNI 192.168.204.148@tcp108 [ 7125.279452] LNet: Removed LNI 192.168.204.148@tcp109 [ 7125.282766] LNetError: 438175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7125.287878] LNetError: 438175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7125.293353] LNet: Removed LNI 192.168.204.148@tcp110 [ 7126.303614] LNet: Removed LNI 192.168.204.148@tcp111 [ 7127.327197] LNet: Removed LNI 192.168.204.148@tcp112 [ 7128.351187] LNet: Removed LNI 192.168.204.148@tcp113 [ 7129.375184] LNet: Removed LNI 192.168.204.148@tcp114 [ 7130.399178] LNet: Removed LNI 192.168.204.148@tcp115 [ 7131.423233] LNet: Removed LNI 192.168.204.148@tcp116 [ 7132.447148] LNet: Removed LNI 192.168.204.148@tcp117 [ 7133.471243] LNet: Removed LNI 192.168.204.148@tcp118 [ 7133.478056] LNetError: 438175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7133.482463] LNetError: 438175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 7133.491262] LNet: Removed LNI 192.168.204.148@tcp119 [ 7134.495156] LNet: Removed LNI 192.168.204.148@tcp120 [ 7135.519280] LNet: Removed LNI 192.168.204.148@tcp121 [ 7136.543955] LNet: Removed LNI 192.168.204.148@tcp122 [ 7137.567163] LNet: Removed LNI 192.168.204.148@tcp123 [ 7138.591327] LNet: Removed LNI 192.168.204.148@tcp124 [ 7139.615255] LNet: Removed LNI 192.168.204.148@tcp125 [ 7140.639174] LNet: Removed LNI 192.168.204.148@tcp126 [ 7140.644990] LNet: Removed LNI 192.168.204.148@tcp127 [ 7141.663177] LNet: Removed LNI 192.168.204.148@tcp128 [ 7142.687647] LNet: Removed LNI 192.168.204.148@tcp129 [ 7142.693150] LNet: Removed LNI 192.168.204.148@tcp130 [ 7142.698817] LNet: Removed LNI 192.168.204.148@tcp131 [ 7142.705773] LNet: Removed LNI 192.168.204.148@tcp132 [ 7142.709908] LNet: Removed LNI 192.168.204.148@tcp133 [ 7142.713602] LNet: Removed LNI 192.168.204.148@tcp134 [ 7142.724766] LNet: Removed LNI 192.168.204.148@tcp135 [ 7143.268564] Key type .llcrypt unregistered [ 7143.271808] Key type ._llcrypt unregistered [ 7159.596322] Key type ._llcrypt registered [ 7159.601057] Key type .llcrypt registered [ 7159.712656] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 7170.707178] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 7171.411115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7171.474947] alg: No test for adler32 (adler32-zlib) [ 7172.449428] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 7172.642477] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 7174.304678] Key type lgssc registered [ 7175.174473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7203.550937] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:38:47 (1775839127) [ 7217.727862] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 7220.037659] Key type lgssc unregistered [ 7220.271131] LNet: 442188:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7220.281086] LNetError: 442188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7220.303201] LNet: Removed LNI 192.168.204.148@tcp [ 7220.953173] Key type .llcrypt unregistered [ 7220.957561] Key type ._llcrypt unregistered [ 7232.715362] Key type ._llcrypt registered [ 7232.719133] Key type .llcrypt registered [ 7232.797047] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 7233.422088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7233.442576] alg: No test for adler32 (adler32-zlib) [ 7234.484364] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 7234.682219] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 7236.344581] Key type lgssc registered [ 7237.280213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7245.287838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7245.304651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7246.524763] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7246.558725] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7246.616488] Lustre: lustre-MDT0000: new disk, initializing [ 7246.676606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7246.693266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7250.035538] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7258.047729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7258.173761] Lustre: 443364:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7258.210412] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7258.213555] Lustre: Skipped 1 previous similar message [ 7258.294343] Lustre: lustre-MDT0001: new disk, initializing [ 7258.417983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7258.465887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7258.486269] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7262.318411] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7269.179836] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7272.875761] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7278.844978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7279.066853] Lustre: lustre-OST0000: new disk, initializing [ 7279.070809] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7279.156409] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7286.155035] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7287.881464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7287.897087] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7288.030230] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7293.050653] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7297.087614] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing lnet_if_list [ 7302.320216] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7302.368697] LNet: Added LNI 192.168.204.148@tcp10 [8/256/0/180] [ 7308.626330] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7308.668321] LNet: Added LNI 192.168.204.148@tcp11 [8/256/0/180] [ 7313.707670] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7313.761549] LNet: Added LNI 192.168.204.148@tcp12 [8/256/0/180] [ 7318.814113] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7318.894143] LNet: Added LNI 192.168.204.148@tcp13 [8/256/0/180] [ 7324.178970] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7324.205769] LNet: Added LNI 192.168.204.148@tcp14 [8/256/0/180] [ 7328.860299] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7328.891928] LNet: Added LNI 192.168.204.148@tcp15 [8/256/0/180] [ 7333.140878] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7333.190449] LNet: Added LNI 192.168.204.148@tcp16 [8/256/0/180] [ 7337.435792] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7337.472931] LNet: Added LNI 192.168.204.148@tcp17 [8/256/0/180] [ 7341.610184] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7341.636738] LNet: Added LNI 192.168.204.148@tcp18 [8/256/0/180] [ 7345.763311] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7345.784209] LNet: Added LNI 192.168.204.148@tcp19 [8/256/0/180] [ 7349.819608] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7349.853927] LNet: Added LNI 192.168.204.148@tcp20 [8/256/0/180] [ 7354.803189] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7354.824766] LNet: Added LNI 192.168.204.148@tcp21 [8/256/0/180] [ 7359.294678] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7359.323532] LNet: Added LNI 192.168.204.148@tcp22 [8/256/0/180] [ 7363.891525] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7363.932531] LNet: Added LNI 192.168.204.148@tcp23 [8/256/0/180] [ 7368.800773] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7368.830394] LNet: Added LNI 192.168.204.148@tcp24 [8/256/0/180] [ 7373.524819] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7373.555347] LNet: Added LNI 192.168.204.148@tcp25 [8/256/0/180] [ 7379.397324] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7379.428227] LNet: Added LNI 192.168.204.148@tcp26 [8/256/0/180] [ 7384.483559] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7384.522294] LNet: Added LNI 192.168.204.148@tcp27 [8/256/0/180] [ 7389.662875] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7389.690289] LNet: Added LNI 192.168.204.148@tcp28 [8/256/0/180] [ 7395.381697] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7395.410922] LNet: Added LNI 192.168.204.148@tcp29 [8/256/0/180] [ 7400.336535] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7400.370387] LNet: Added LNI 192.168.204.148@tcp30 [8/256/0/180] [ 7405.411461] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7405.450512] LNet: Added LNI 192.168.204.148@tcp31 [8/256/0/180] [ 7410.661343] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7410.687244] LNet: Added LNI 192.168.204.148@tcp32 [8/256/0/180] [ 7415.565406] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7415.602259] LNet: Added LNI 192.168.204.148@tcp33 [8/256/0/180] [ 7420.879827] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7420.902710] LNet: Added LNI 192.168.204.148@tcp34 [8/256/0/180] [ 7425.960833] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7425.982743] LNet: Added LNI 192.168.204.148@tcp35 [8/256/0/180] [ 7431.330612] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7431.359346] LNet: Added LNI 192.168.204.148@tcp36 [8/256/0/180] [ 7436.961263] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7436.982725] LNet: Added LNI 192.168.204.148@tcp37 [8/256/0/180] [ 7443.386363] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7443.408597] LNet: Added LNI 192.168.204.148@tcp38 [8/256/0/180] [ 7453.781961] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7453.813740] LNet: Added LNI 192.168.204.148@tcp39 [8/256/0/180] [ 7465.209397] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7465.286026] LNet: Added LNI 192.168.204.148@tcp40 [8/256/0/180] [ 7476.842291] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7476.899362] LNet: Added LNI 192.168.204.148@tcp41 [8/256/0/180] [ 7492.820441] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7492.898210] LNet: Added LNI 192.168.204.148@tcp42 [8/256/0/180] [ 7501.780136] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7501.815737] LNet: Added LNI 192.168.204.148@tcp43 [8/256/0/180] [ 7509.958445] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7509.981072] LNet: Added LNI 192.168.204.148@tcp44 [8/256/0/180] [ 7518.828639] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7518.870462] LNet: Added LNI 192.168.204.148@tcp45 [8/256/0/180] [ 7526.913522] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7526.966976] LNet: Added LNI 192.168.204.148@tcp46 [8/256/0/180] [ 7535.138843] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7535.162955] LNet: Added LNI 192.168.204.148@tcp47 [8/256/0/180] [ 7542.885703] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7542.955544] LNet: Added LNI 192.168.204.148@tcp48 [8/256/0/180] [ 7553.569228] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7553.695080] LNet: Added LNI 192.168.204.148@tcp49 [8/256/0/180] [ 7566.614511] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7566.655563] LNet: Added LNI 192.168.204.148@tcp50 [8/256/0/180] [ 7590.881347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7590.886889] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7590.909973] Lustre: Skipped 2 previous similar messages [ 7590.934761] Lustre: Skipped 2 previous similar messages [ 7596.009358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7596.014733] Lustre: Skipped 2 previous similar messages [ 7596.650808] Lustre: server umount lustre-MDT0000 complete [ 7601.818973] LustreError: 444140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775839527 with bad export cookie 1755431534349874155 [ 7601.832251] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7601.836778] LustreError: 444140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7602.261504] Lustre: server umount lustre-MDT0001 complete [ 7621.606048] Lustre: 442634:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775839531/real 1775839531] req@0000000043654f5b x1862102323204736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775839547 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7621.635072] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7622.989418] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 7626.719698] Lustre: 442636:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775839536/real 1775839536] req@000000005241ca0e x1862102323204864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775839552 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7627.745179] Lustre: 442637:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775839537/real 1775839537] req@0000000007297ed2 x1862102323205120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775839553 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7632.388435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7633.888296] Lustre: 442635:0:(client.c:2479:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1775839543/real 1775839543] req@00000000d47e14cb x1862102323205248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1775839559 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7647.595809] LustreError: 445098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7647.614554] LustreError: 445098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7647.705638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7652.543267] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7652.834739] LustreError: 445099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7655.905636] LustreError: 445098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7655.908272] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7661.030864] LustreError: 445099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7661.809839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7662.361883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7666.489975] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7667.062855] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7673.539846] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7678.657622] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7702.501341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7702.505724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7702.529433] Lustre: Skipped 1 previous similar message [ 7702.561850] Lustre: Skipped 1 previous similar message [ 7707.622649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7712.736862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7712.744163] Lustre: Skipped 2 previous similar messages [ 7716.320547] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7716.645272] Lustre: server umount lustre-OST0000 complete [ 7722.984083] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7722.990631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7723.004642] Lustre: Skipped 1 previous similar message [ 7723.023379] Lustre: Skipped 1 previous similar message [ 7727.833805] Lustre: server umount lustre-MDT0000 complete [ 7728.103053] LustreError: 468632:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7728.133054] LustreError: 468632:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7732.656743] LustreError: 443356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775839658 with bad export cookie 1755431534349876731 [ 7732.659250] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7732.673379] LustreError: 443356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7733.402336] Lustre: server umount lustre-MDT0001 complete [ 7745.916812] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 7749.281404] Key type lgssc unregistered [ 7749.850103] LNet: 471190:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7749.866756] LNetError: 471190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7749.872128] LNet: Removed LNI 192.168.204.148@tcp [ 7749.875812] LNet: Removed LNI 192.168.204.148@tcp10 [ 7749.883692] LNet: Removed LNI 192.168.204.148@tcp11 [ 7749.889718] LNet: Removed LNI 192.168.204.148@tcp12 [ 7749.894258] LNet: Removed LNI 192.168.204.148@tcp13 [ 7749.897701] LNet: Removed LNI 192.168.204.148@tcp14 [ 7749.899915] LNet: Removed LNI 192.168.204.148@tcp15 [ 7749.906834] LNet: Removed LNI 192.168.204.148@tcp16 [ 7749.909614] LNet: Removed LNI 192.168.204.148@tcp17 [ 7749.911505] LNet: Removed LNI 192.168.204.148@tcp18 [ 7749.915656] LNet: Removed LNI 192.168.204.148@tcp19 [ 7749.923425] LNet: Removed LNI 192.168.204.148@tcp20 [ 7749.927691] LNet: Removed LNI 192.168.204.148@tcp21 [ 7749.932314] LNet: Removed LNI 192.168.204.148@tcp22 [ 7749.939865] LNet: Removed LNI 192.168.204.148@tcp23 [ 7749.942974] LNet: Removed LNI 192.168.204.148@tcp24 [ 7749.945846] LNet: Removed LNI 192.168.204.148@tcp25 [ 7749.950687] LNet: Removed LNI 192.168.204.148@tcp26 [ 7749.956184] LNet: Removed LNI 192.168.204.148@tcp27 [ 7749.963063] LNet: Removed LNI 192.168.204.148@tcp28 [ 7749.965286] LNet: Removed LNI 192.168.204.148@tcp29 [ 7749.977105] LNet: Removed LNI 192.168.204.148@tcp30 [ 7749.979248] LNet: Removed LNI 192.168.204.148@tcp31 [ 7749.988053] LNet: Removed LNI 192.168.204.148@tcp32 [ 7749.994859] LNet: Removed LNI 192.168.204.148@tcp33 [ 7750.006681] LNet: Removed LNI 192.168.204.148@tcp34 [ 7750.012830] LNet: Removed LNI 192.168.204.148@tcp35 [ 7750.020097] LNet: Removed LNI 192.168.204.148@tcp36 [ 7750.036817] LNet: Removed LNI 192.168.204.148@tcp37 [ 7750.045185] LNet: Removed LNI 192.168.204.148@tcp38 [ 7750.054860] LNet: Removed LNI 192.168.204.148@tcp39 [ 7750.060130] LNet: Removed LNI 192.168.204.148@tcp40 [ 7750.064675] LNet: Removed LNI 192.168.204.148@tcp41 [ 7750.070651] LNet: Removed LNI 192.168.204.148@tcp42 [ 7750.074696] LNet: Removed LNI 192.168.204.148@tcp43 [ 7750.078281] LNet: Removed LNI 192.168.204.148@tcp44 [ 7750.085428] LNet: Removed LNI 192.168.204.148@tcp45 [ 7750.091625] LNet: Removed LNI 192.168.204.148@tcp46 [ 7750.098159] LNet: Removed LNI 192.168.204.148@tcp47 [ 7750.103099] LNet: Removed LNI 192.168.204.148@tcp48 [ 7750.107431] LNet: Removed LNI 192.168.204.148@tcp49 [ 7750.131608] LNet: Removed LNI 192.168.204.148@tcp50 [ 7751.142556] Key type .llcrypt unregistered [ 7751.148196] Key type ._llcrypt unregistered [ 7775.669440] Key type ._llcrypt registered [ 7775.672748] Key type .llcrypt registered [ 7775.888172] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 7777.262483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7777.486157] alg: No test for adler32 (adler32-zlib) [ 7778.825647] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 7779.111645] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 7780.865534] Key type lgssc registered [ 7782.295484] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7792.732459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7792.770472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7794.525995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7799.995915] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7804.899433] LustreError: 472435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7804.933842] LustreError: 472435:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7810.026358] LustreError: 472436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7811.557482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7812.127734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7817.711521] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7825.347386] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7830.867808] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7839.238675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7839.606086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7840.657225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7847.839207] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7855.476320] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7866.395873] loop0: detected capacity change from 0 to 204800000 [ 7866.611727] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7866.636744] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7866.672289] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7873.643751] loop0: detected capacity change from 0 to 204800000 [ 7873.856929] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7873.871703] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7873.889508] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7879.354024] loop0: detected capacity change from 0 to 204800000 [ 7885.256123] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7885.401288] Lustre: fs15246-OST0000: new disk, initializing [ 7885.407951] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7885.457507] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7891.889079] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7901.711961] loop1: detected capacity change from 0 to 204800000 [ 7908.111236] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7908.234586] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7908.301948] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7908.387379] Lustre: fs15246-MDT0000: new disk, initializing [ 7908.588378] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7908.623543] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7908.734178] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7909.793357] Lustre: 476803:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000cef2ac28 x1862102893510272/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1775839851 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7913.240109] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 7914.878884] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 7914.903805] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 7914.948196] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 7939.565174] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7939.579560] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7941.268565] Lustre: server umount fs15246-MDT0000 complete [ 7944.679217] LustreError: 472431:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7949.796622] LustreError: 472431:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7950.530550] Lustre: server umount fs15246-OST0000 complete [ 7963.622675] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7963.625469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7963.636642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7964.640570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7964.655693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7964.672835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7966.640573] Lustre: server umount lustre-OST0000 complete [ 7970.273762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7970.292193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7970.294787] Lustre: Skipped 2 previous similar messages [ 7975.392836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7975.407114] Lustre: Skipped 1 previous similar message [ 7976.227839] Lustre: server umount lustre-MDT0000 complete [ 7979.977650] LustreError: 477833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775839905 with bad export cookie 16986573688797750283 [ 7979.984529] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7979.996643] LustreError: 477833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7980.366932] Lustre: server umount lustre-MDT0001 complete [ 7990.218622] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 7992.629689] Key type lgssc unregistered [ 7992.883147] LNet: 479869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7992.889152] LNetError: 479869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7992.904931] LNet: Removed LNI 192.168.204.148@tcp [ 7993.582647] Key type .llcrypt unregistered [ 7993.586672] Key type ._llcrypt unregistered [ 8009.022949] Key type ._llcrypt registered [ 8009.026541] Key type .llcrypt registered [ 8009.128822] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 8010.292448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8010.323675] alg: No test for adler32 (adler32-zlib) [ 8011.314054] Lustre: Lustre: Build Version: 2.17.51_75_gd106f95 [ 8011.478808] LNet: Added LNI 192.168.204.148@tcp [8/256/0/180] [ 8013.119174] Key type lgssc registered [ 8014.155511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8047.369860] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 8056.779305] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 8085.814657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8085.864129] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8087.097299] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8087.142711] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8087.250127] Lustre: lustre-MDT0000: new disk, initializing [ 8087.335869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8087.352428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8090.291197] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8098.936776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8099.047986] Lustre: 484261:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8099.096645] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8099.100582] Lustre: Skipped 1 previous similar message [ 8099.149932] Lustre: lustre-MDT0001: new disk, initializing [ 8099.242299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8099.270736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8099.284593] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8102.937117] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8108.990695] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8112.068469] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8117.887707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8118.099852] Lustre: lustre-OST0000: new disk, initializing [ 8118.103782] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8118.190118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8123.734858] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_default_debug -1 all [ 8126.513619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8126.520729] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8126.623560] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8129.604706] Lustre: DEBUG MARKER: oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8133.270158] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8133.463463] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8136.672617] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8136.860614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8141.796224] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8141.805561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8141.833347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8142.310967] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8144.490282] Lustre: server umount lustre-OST0000 complete [ 8150.496358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8150.503627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8150.524813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8150.530174] Lustre: Skipped 2 previous similar messages [ 8152.546438] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8152.547508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8152.570731] Lustre: Skipped 1 previous similar message [ 8153.999618] Lustre: server umount lustre-MDT0000 complete [ 8156.981694] LustreError: 484255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1775840082 with bad export cookie 15159397241908866014 [ 8156.987568] LustreError: MGC192.168.204.148@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8156.999410] LustreError: 484255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8157.274787] Lustre: server umount lustre-MDT0001 complete [ 8175.848502] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing set_hostid [ 8183.453602] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing load_modules_local [ 8207.463913] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7940 sec ========== 12:55:31 (1775840131) [ 8208.808282] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:55:32 (1775840132) === [ 8211.332526] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:55:35 (1775840135) === [ 8241.540685] Lustre: DEBUG MARKER: oleg448-server.virtnet: executing unload_modules_local [ 8243.562258] Key type lgssc unregistered [ 8243.835788] LNet: 492445:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8243.842362] LNetError: 492445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8243.859688] LNet: Removed LNI 192.168.204.148@tcp [ 8244.565379] Key type .llcrypt unregistered [ 8244.568414] Key type ._llcrypt unregistered